Lecture Manual
Transcription
Lecture Manual
Lecture Manual Introduction to ACE Analyst NOTICES Product Information Lecture Manual for: ACE Analyst Software Release: 16.0.T PL0 Destination Control Statement This Lab Manual and the OPNET Software were manufactured in the United States by OPNET. The initial export of this Lab Manual and the OPNET Software, and any subsequent relocation to another country shall comply with United States law relating to the export of technical data, equipment, software and know-how. Technical Support Restricted Rights Legend Telephone: 240.497.1200 Facsimile: 240.497.1064 E-mail: [email protected] Web: http://www.opnet.com OPNET Technologies, Inc. (“OPNET”) is the sole owner of all proprietary rights in this Lab Manual (the “Lab Manual”) and the OPNET IT Guru software (the “OPNET Software”). Terms and Conditions of Use Access to and use of the Training Manual and/or the OPNET Software is restricted to authorized users who are employees or direct contractors of a Licensee who has accepted the terms and conditions of OPNET’s Software Master License Agreement. As a condition of having access to or use of the Lab Manual and/or the OPNET Software all users agree to be subject the terms of OPNET’s Software Master License Agreement. Users shall not copy or reproduce, in whole or in part, or permit access to or disclose the Lab Manual or the OPNET Software to any other person without the prior written consent of OPNET. Liability Statement The terms and conditions governing the licensing of OPNET Software consist of those set forth in the written contracts and license agreements between OPNET and its customers. No representations in this Lab Manual or statements of capability, suitability for use, accuracy or performance, shall be a warranty by OPNET unless contained in a written agreement signed by OPNET. OPNET DISCLAIMS ALL EXPRESS OR IMPLIED WARRANTIES AS TO THE LAB MANUAL AND THE OPNET SOFTWARE INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. USE BY AGENCIES OF THE UNITED STATES GOVERNMENT: For purposes of use of this Lab Manual and OPNET Software by agencies of the United States government (or those authorized to use of this Lab Manual and OPNET Software on behalf of an agency), the OPNET Software is “Commercial Computer Software” within the meaning of DFARs 227.7202 and 252.227-7014(a)(1) and FARs 52.227-19. Use, reproduction and disclosure is subject to DFAR and FAR provisions relating to Commercial Computer Software and the License Agreement between OPNET and the agencies using OPNET Software. OPNET software is provided with a RESTRICTED RIGHTS NOTICE for government use. Trademarks OPNET, OPNET Technologies, Inc., OPNET Modeler, IT Guru, OPNET Development Kit, NetBiz and IT Guru are trademarks of OPNET Technologies, Inc. All other trademarks used in the Lab Manual are the property of their respective owners and used herein for identification purposes only. CONFIDENTIAL INFORMATION DO NOT DISCLOSE, FORWARD, DISTRIBUTE, SHARE, OR MAKE COPIES OF THIS DOCUMENT IN WHOLE OR IN PART. This document contains confidential information and may contain information that is proprietary, privileged, and/or exempt from disclosure under applicable law. This document is intended for the exclusive use of the person to whom it is disclosed. If you are an unauthorized person, you are hereby notified that any viewing, copying, disclosure or distribution of this information may be subject to legal action. All unauthorized persons must immediately destroy the original documentation without making any copies or further unauthorized disclosure. Lecture Manual Introduction to ACE Analyst NOTICES Product Information Lecture Manual for: ACE Analyst Software Release: 16.0.T PL0 Destination Control Statement This Lab Manual and the OPNET Software were manufactured in the United States by OPNET. The initial export of this Lab Manual and the OPNET Software, and any subsequent relocation to another country shall comply with United States law relating to the export of technical data, equipment, software and know-how. Technical Support Restricted Rights Legend Telephone: 240.497.1200 Facsimile: 240.497.1064 E-mail: [email protected] Web: http://www.opnet.com OPNET Technologies, Inc. (“OPNET”) is the sole owner of all proprietary rights in this Lab Manual (the “Lab Manual”) and the OPNET IT Guru software (the “OPNET Software”). Terms and Conditions of Use Access to and use of the Training Manual and/or the OPNET Software is restricted to authorized users who are employees or direct contractors of a Licensee who has accepted the terms and conditions of OPNET’s Software Master License Agreement. As a condition of having access to or use of the Lab Manual and/or the OPNET Software all users agree to be subject the terms of OPNET’s Software Master License Agreement. Users shall not copy or reproduce, in whole or in part, or permit access to or disclose the Lab Manual or the OPNET Software to any other person without the prior written consent of OPNET. Liability Statement The terms and conditions governing the licensing of OPNET Software consist of those set forth in the written contracts and license agreements between OPNET and its customers. No representations in this Lab Manual or statements of capability, suitability for use, accuracy or performance, shall be a warranty by OPNET unless contained in a written agreement signed by OPNET. OPNET DISCLAIMS ALL EXPRESS OR IMPLIED WARRANTIES AS TO THE LAB MANUAL AND THE OPNET SOFTWARE INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. USE BY AGENCIES OF THE UNITED STATES GOVERNMENT: For purposes of use of this Lab Manual and OPNET Software by agencies of the United States government (or those authorized to use of this Lab Manual and OPNET Software on behalf of an agency), the OPNET Software is “Commercial Computer Software” within the meaning of DFARs 227.7202 and 252.227-7014(a)(1) and FARs 52.227-19. Use, reproduction and disclosure is subject to DFAR and FAR provisions relating to Commercial Computer Software and the License Agreement between OPNET and the agencies using OPNET Software. OPNET software is provided with a RESTRICTED RIGHTS NOTICE for government use. Trademarks OPNET, OPNET Technologies, Inc., OPNET Modeler, IT Guru, OPNET Development Kit, NetBiz and IT Guru are trademarks of OPNET Technologies, Inc. All other trademarks used in the Lab Manual are the property of their respective owners and used herein for identification purposes only. CONFIDENTIAL INFORMATION DO NOT DISCLOSE, FORWARD, DISTRIBUTE, SHARE, OR MAKE COPIES OF THIS DOCUMENT IN WHOLE OR IN PART. This document contains confidential information and may contain information that is proprietary, privileged, and/or exempt from disclosure under applicable law. This document is intended for the exclusive use of the person to whom it is disclosed. If you are an unauthorized person, you are hereby notified that any viewing, copying, disclosure or distribution of this information may be subject to legal action. All unauthorized persons must immediately destroy the original documentation without making any copies or further unauthorized disclosure. 2/10/2011 Introduction to AppTransaction Xpert (ACE Analyst) test Day 1 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. About OPNET Technologies, Inc.® Corporate Overview • Founded in 1986 • Publicly traded (NASDAQ: OPNT) • HQ in Bethesda, MD • Approximately 600 employees • Worldwide presence through direct offices and channel partners • Global distribution agreement with Cisco Best-in-Class Solutions and Services • Application Performance Management • Network Performance Management • Network R&D Strong Financial Track Record g historyy of profitability p y • Long • Trailing 12-month revenue of over $100M • Approximatelytest 25% of revenue re-invested in R&D Broad Customer Base • Corporate Enterprises • Government Agencies/DoD • Service Providers • Network Equipment Manufacturers 3 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 1 2/10/2011 Customers (partial list) Financial Services Government Insurance Healthcare / Bio / Pharmaceutical Manufacturing Energy / Utilities Media / Communications Services test Retail / Consumer Products 4 Transportation / Logistics OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET’s APM Xpert Suite End-user experience and network monitoring and analytics Deep application component monitoring and analytics Deep transaction tracing, analysis, and prediction Passive database performance monitoring test Network and server discovery and monitoring “What's in your Application?” 5 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 2 2/10/2011 OPNET is an IACET Authorized Provider OPNET Technologies, Inc. has been approved as an Authorized Provider by the International Association for Continuing Education and Training (IACET), 1760 Old Meadow Road, Suite 500, McLean, VA 22102. In obtaining this approval, OPNET has demonstrated that it complies with the ANSI/IACET 11-2007 2007 Standard which is internationally recognized as the Standard for training organizations. As a result of their Authorized Provider status OPNET is authorized to offer IACET Continuing Education Units (CEUs) for its programs that qualify under the ANSI/IACET 1-2007 Standard. test 6 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Continuing Education Units What is IACET? IACET is a Standards Developing Organization accredited by the American Standards Institute (ANSI) and is an authorizing body for education providers that meet the American National Standard for continuing education and training (ANSI/IACET 1-2007 Standard). The IACET standard is what learners seek for quality and is the core of thousands of educational programs worldwide. (see: www.IACET.org) What is an Authorized Provider? The IACET Authorized Provider (AP) status means that an organization adheres to IACET’s internationally recognized ANSI/IACET 1-2007 Standard for Continuing Education and Training. Only Authorized Providers may award IACET Continuing Education Units (CEUs). Whatt is Wh i a CEU? test The Continuing Education Unit (CEU) was created by IACET as a measurement of continuing education. One (1) IACET CEU is equal to ten (10) contact hours of participation in an organized continuing education experience under responsible sponsorship, capable direction, and qualified instruction. 7 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 3 2/10/2011 Earning CEUs for this Course Introduction to AppTransaction Xpert – This course is worth 1.3 CEUs • To receive these CEUs, you must – – – Attend the entire course Fill out the feedback form Score 80% in a test at the end of class • Your test is scored in the next 10 business days • If successful, you will receive a certificate denoting your CEUs in the mail within 2 weeks test of the course is stored in our database for later retrieval as • Your completion needed 8 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Let’s get to know each other… • Name ? • What company you work for ? • Which Department ? • What experience do you have with OPNET products? • Did you know we changed the names… !!?? test 9 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 4 2/10/2011 Course Structure 10 Day 1 Introduction to AppTransaction Xpert Day 2 Advanced AppTransaction Xpert •About OPNET •Visualizations in AppTransaction Xpert •Troubleshooting application performance •Predicting application behavior before deployment test strategies •Capture and import •WAN Acceleration •Advanced Topics •Understanding Delay Components •Parallel Effects, User think Time •Troubleshooting TCP applications OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Class Format • In general, four phase instruction – Short lecture sections – Demonstrations – Related hands-on labs – Discussion of lab applications • Class is interactive – don’t hesitate to ask questions • Please fill out the feedback form online @ eval.opnet.com – Give us information on your current network and application concerns test – Provide suggestions on how we can improve our products – Let us know how we can improve this training class 11 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 5 2/10/2011 Agenda Introduction - Licensing Options - Overview, Use Cases - Basic Workflow Using AppTransaction Xpert to visualize applications - AppTransaction Xpert Visualizations - Lab 1: Visualizing Application Using AppTransaction Xpert to diagnose problematic applications - AppDoctor Reporting capabilities of AppTransaction Xpert - Reports R t test - Lab 2: Application Performance Troubleshooting Using AppTransaction Xpert to perform performance predictive studies - QuickPredict and QuickRecode - Lab 3: Predictive Analysis 12 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Licensing Options • Download and administer licenses using the License Manager • Licensed on concurrent user basis • Online license management g (http://www.opnet.com/support/license_management.html) • Licensing options – Standalone vs. Floating (license server) – Loanable Licenses test 13 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 6 2/10/2011 Overview of AppTransaction Xpert test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. AppTransaction Xpert Solution • AppTransaction Xpert Plus • AppTransaction Xpert Standard • IT Guru with the following modules – ACE – ACE Decode Module – Advanced ACE Capabilities • Starting with version 16.0, the different offerings of AppTransaction Xpert (that is: AppTransaction Xpert Plus, AppTransaction Xpert Standard, and AppTransaction Xpert Module) have been combined into a single installer. test Important—Existing users are not losing any AppTransaction Xpert functionality because of this change. Functionality is determined by licenses. 15 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 7 2/10/2011 What is AppTransaction Xpert... ? • Application Characterization Environment • Capture application traces • Visualize Vi li Application A li ti Dynamics D i – Application layer analysis – Network layer analysis • Diagnose performance problems – Minimize troubleshooting time • Validate solutions test – Predict application behavior under different network conditions – Evaluate performance impact of application design decisions – Validate infrastructure readiness for new deployments • Generate reports 16 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. When to use AppTransaction Xpert ?? (Use Cases) test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 8 2/10/2011 Use Case 1 Troubleshooting application performance – • Users of a critical application are reporting very bad response times – Network engineers are reporting no issues on the network – Application engineers are reporting no issues with software • Result: Finger pointing !! Wasted time test 18 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Use Case 2 Testing changes of the enterprise environment – • Relocation of key processing data center from US to China • Management needs to know the implications of move – Impact on application performance and network utilization? – Will additional resources be required? • Result: Need to produce accurate performance projections and network planning l i requirements i test 19 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 9 2/10/2011 Use Case 3 Application pre-deployment analysis – • Purchase of a new 3-party application system • Management needs to know how the new system will perform – Application response times – Network utilization • Result: Need to eliminate the need for trial and error deployment of application. Expensive and time consuming. test 20 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Basic Workflow test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 10 2/10/2011 What does AppTransaction Xpert tell you … ? • “Why is my application slow?” – – – – – – User’s desktop running slow? Server delays? y Non-optimal protocol settings (TCP and others)? Heavy congestion over WAN? Too little bandwidth? Application too chatty? • “Can this new application be deployed over our network?” – Expected test transaction response times? – Impact other critical applications? – Bandwidth requirements? 22 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. How does AppTransaction Xpert tell you this...? • Reports, graphs, and tables • Visualization of application pp behavior • Diagnosis of bottlenecks and performance problems • Predictive analysis of same traffic on different network • Input to other OPNET products for further analysis – Application traffic analysis test – Capacity planning studies 23 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 11 2/10/2011 Inputs to AppTransaction Xpert • Packet traces from the following sources – ACE Capture Agents – ACE Live – Cisco NAM Blade – WAN Accelerator – F5 BIG-IP – Sniffer – UNIX (no agent installed) – Tcpdump, windump ..etc • Basic characteristics of network (Recommended) test – Tier locations – Bandwidth between tiers – Latency between tiers 24 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Application Troubleshooting Workflow Identify performance issues Capture Packet Traces Visualize, Diagnose, and Predict Optional: Detailed Analysis test Optional: Create IT Guru network model Generate Reports 25 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 12 2/10/2011 AppTransaction Xpert Visualizations test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. AppTransaction Xpert Visualizations • Data Exchange Chart – Shows timeline of data sent • Tier Pair Circle – Shows summary statistics • Tree View – Shows Transaction Analysis • Decodes – Shows Application layer headers • Graphs test – View time-based graphs of application statistics 27 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 13 2/10/2011 Data Exchange Chart (DEC) • Timeline of application messages and network packets between tiers • Slope indicates time to cross network • Color indicates payload size of message test 28 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. DEC: Message Groups • When messages and packet are too frequent to be displayed individually, a solid block is shown • Block is a histogram showing the size of messages in groups • Zoom in or use tooltip to see more detail test 29 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 14 2/10/2011 DEC: View Options •Application Chart Only Shows every application message – several packets may comprise a message • Network Chart Only Shows every individual packet in the trace file, including TCP acknowledgements • Application and Network Charts Shows both charts in the same window test Application Messages Network Packets 30 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. DEC: Show Dependencies • Triangles appear at largest chunks of delay Click “Show Dependencies” • White = delay at host • Orange = user think time • Purple = network delay test Legend 31 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 15 2/10/2011 DEC: Application Characteristics • Quiet Time – – – – User think time? Slow client computer? p Slow server? Not seeing the whole picture? §Tip: See what happened before and f silence after test Tip: Check Network Chart for more details Large periods of silence 32 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. DEC: Application Characteristics (cont.) • Chatty Application – Lots of back-and-forth communication (Application Turns) Lots of small messages (orange and yellow) test Groups of messages close together 33 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 16 2/10/2011 Application Turn • Counted on the Application Layer (Application Messages) • Every turn has to cross the network latency which adds up to the total Latency Delay 2 Client Network Latency test Server 1 34 3 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. DEC: Application Characteristics • One-way data transfers – Dark blue arrows – Most arrows in same direction Tip: Check the Network Packet Chart for more details test Zoom to see the arrow heads 35 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 17 2/10/2011 DEC: Application Characteristics (cont.) • Lots of irrelevant traffic – Do not try diagnose application until you removed irrelevant data test Tip: Go to Tier Pair Circle and delete unrelated traffic 36 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Tier Pair Circle • Summary of traffic between all tiers g – A high-level view of the application task – Is there extraneous traffic? • Right-click to exclude unwanted traffic test Tip: If you don’t see the protocol you expect, then right-click to re-recode traffic on the fly 37 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 18 2/10/2011 Tier Pair Circle: Functionality • Choose statistics to view • Right-click a tier or conversation to exclude /delete it or all others • Show only traffic containing a specific protocol • Find Top p Statistics to sort conversations by relevance test 38 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Tier Pair Circle: Statistics • Pull down to select statistics to view: test 39 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 19 2/10/2011 Tree View • Tree pane: One-line summary of tier pairs, connections and frames • Table pane: Detail statistics by tier pairs, connections and frames • Timeline ppane: Overview of messages g over the time of the task test Tree Pane 40 Table Pane Timeline Pane OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Tree View: “View by:” Options • Tier Pairs – Network Packets • Tier Pairs – Application Transactions • Application Transactions Group by: test 41 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 20 2/10/2011 Tree View: Sorting • Click a column header to sort the table by descending values. • Click the table header again to sort by ascending values. test 42 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Tree View : Customize the Table Pane • Choose View > Choose Table Columns… • Right-click on the column header and click “Choose TreeView Columns…” • Select the network statistics to appear in the table test 43 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 21 2/10/2011 Decode Viewer ETH IP TCP HTTP ETH • What are decodes? – Analysis or description of the bytes at each layer of a packet • What can you test learn from decodes? Detailed protocol information Information about data being transferred Ex: TCP gives Sequence number and Acknowledgement number 44 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Embedded Decode Viewer • Show / Hide the embedded decode viewer by clicking and • Select traffic in the top pane and see corresponding decodes in the bottom pane • Right-click on selected frames to export to test spreadsheet • Click the plus sign to expand layers of decodes 45 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 22 2/10/2011 Tree View to DEC • Display Context-Based Traffic in Data Exchange Chart — – Double-click on traffic for a tier pair in the Timeline pane to display the corresponding context- based traffic in the Data Exchange Chart. test 46 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Graphs • Time varying statistics – Advanced diagnosis of network and TCP problems test 47 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 23 2/10/2011 Graphs (cont.) • Select graphs to display • Options: – Specify bucket width for precision of graph – Embed graphs in DEC for easy comparison against messages in DEC test – Overlay similar graphs or overlay all • Select “Show” 48 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Graphs: Bucket Width • What is a bucket width? – Statistics can be averaged over time rather than shown as individual data points – A bucket is the duration of data you want to average, like 100 ms or 3 seconds • Same trace, same statistics, d e e t bucket different buc et widths: w dt s: – 100 ms test – 3000 ms (3 seconds) 49 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 24 2/10/2011 Graphs: Embedded Statistics • Time-aligned statistics with the Data Exchange Chart test 50 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. LAB 1 LAB 1 Visualizing Application test 51 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 25 2/10/2011 Lab 1: Visualizing Application • Overview – The ACE trace file represents p a three-tier Oracle application. pp You will explore some of the different visualizations in AppTransaction Xpert that you can use to troubleshoot and analyze the application. • Starts on page 1 in your lab manual test 52 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Lab 1: Conclusion AppTransaction Xpert provides many different views of a transaction to understand the interaction of the application on the network • Data Exchange Chart – Shows timeline of data sent • Tier Pair Circle – Shows who talks to whom and summary statistics • Tree View – View the network packets in order, with a decode summary – Transaction analysis • See the timeline of transactions • Decodes – Drill downtest into TCP, IP, or even application layer headers to understand application behavior • Graphs – View time-based graphs of throughput, congestion, and other statistics 53 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 26 2/10/2011 AppDoctor test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. What is AppDoctor … ? • Invoke AppDoctor – AppDoctor (menu) – AppDoctor (icon) • AppDoctor Tabs – Summary of Delays – Executive Summary – Diagnosis – Statistics – Protocol-specific test 55 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 27 2/10/2011 AppDoctor: Summary of Delays • Pie chart that breaks down the total response time into processing, network, and parallel ll l effects ff t • Most useful view in AppTransaction Xpert when troubleshooting applications • Quickly points you to what is the problem, or at least what is test definitely not the problem • Must be used on a single application task 56 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Summary of Delays: Processing Effects • Tier Processing delay – Time spent processing on each tier test 57 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 28 2/10/2011 Summary of Delays: Network Effects • Latency – Delay caused by the physical distance between tiers – Common p problem in chattyy applications • Bandwidth – Delay caused by transmission (i.e. putting the bits onto the wire) • Protocol – Could be non-optimal TCP settings – Check Diagnosis tab for bottleneckstest – Check Statistics tab for values • Congestion – Delay caused by other traffic in the network 58 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Summary of Delays: Parallel Effects • Delay incurred when multiple types of delay (such as tier processing, bandwidth, and latency) occur simultaneously test 59 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 29 2/10/2011 AppDoctor – After Network Definition (16.0 ) test BEFORE 60 AFTER OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Summary of Delays: Drill-Downs • Double-click any section of the pie chart for extra information • Drill-downs give more information that can help diagnose problems • Ex: Bandwidth drill-down shows throughput g p graphs g p and link bandwidth test 61 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 30 2/10/2011 AppDoctor: Executive Summary • Gives overview information for application task – Total transaction time – Total application data transferred – Single Si l greatest t t cause off delay d l – Tiers in application – Bandwidth and latency at each tier test 62 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. AppDoctor: Diagnosis • Displays common application, network, and server problems • Flags bottlenecks and potential bottlenecks • Various characteristics of the ACE file are compared with thresholds • Click on any cell (especially on test bottlenecks) for extra information 63 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 31 2/10/2011 AppDoctor: Statistics • Tabular view of key application characteristics of” statistics quantify • “Effect Effect of the components of delay • Other statistics include – Application message statistics – Network packet statistics – TCP statistics Tip: Use it with Diagnosis tab to test quantify bottlenecks Tip: You can export to spreadsheet 64 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. AppDoctor: Protocol-Specific Tabs • Protocol-specific information, statistics, and diagnoses • Appear automatically when certain protocols are detected in an ACE file: – HTTP – Citrix – CORBA – SOAP (Web Services) test 65 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 32 2/10/2011 Right-Click Operations in AppTransaction Xpert • Graph Statistics such as throughput or congestion • Exclude or permanently delete packets and messages • Include ppackets and messages g • Rename a tier or lookup a tier name with DNS • Re-code traffic • Zoom in and out in the DEC • Show protocol decodes for selected traffic • Export decodes for selected frames in the decode viewer test 66 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Take-Away Points AppTransaction Xpert provides an analysis of the main application bottlenecks that is easy to use and to understand. The diagnosis is displayed and broken down into components of delay. The different components are: • Processing Effects • Network Effects – – – – Latency Bandwidth Protocol C Congestion ti (with recommended options) test • Other Effects – Parallel Effects: Combination of any processing effects and network effects that occur at the same time 67 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 33 2/10/2011 Reports test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Reporting Motivation • Quick and easy to use • Support your findings • Recording and dissemination of all analysis results to all interested stake holders • Documentation of application behavior for future reference test 69 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 34 2/10/2011 Report Selection - Type and Format • Very simple and quick – select report type for output (from pull down menu, or icon button) test 70 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Report Options • Select report options and generate report – Report sections, title (statistics) – Report directory (output), Report Server (optional) test 71 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 35 2/10/2011 ACE – MS Word Report test 72 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. ACE – Web Report URL Home Dir (all reports) test 73 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 36 2/10/2011 ACE – Multiple Transaction Report Trans 1 Trans 2 test 74 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. ACE – Difference Report test 75 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 37 2/10/2011 ACE – Spreadsheet Export Reports Comparison Report Tree View Statistics AppDoctor Statistics test 76 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. ACE – Report Template • Report template — Defines the previously-defined templates used to generate the report. You can modify and customize report templates. The standard templates: – ApplicationDeveloperReport.ace.rt.txt – ManagementReport.ace.rt.txt – NetworkEngineerReport.ace.rt.txt – concise_report.ace.rt.txt – detailed_report_ace.rt.txt – standard_report_ace.rt.txt • Reports can be cataloged and managed by the Report Server test – Provides security, allows access by only registered users – Provides automatic ageing of reports 77 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 38 2/10/2011 LAB 2 LAB 2 Application Performance Troubleshooting test 78 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Lab 2: Application Performance Troubleshooting • Overview – Response time problems for remote users of new web application – Remote site connected via a 256-Kbps Frame Relay circuit – Need to determine cause of the delay (network or web server) – Traces have been captured at both server and client • Objectives – Visualize application in ACE – Identify the cause of the application delay – Use Transaction Analysis to identify problem with application • Starts on page 9 in your lab manual test 79 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 39 2/10/2011 Lab 2: Conclusion • Conclusions – – – – – – – 80 The web page takes approximately 24 seconds to download Bandwidth and congestion are the biggest sources of delay Server delay is not a problem Majority of traffic flow is from the server to the client Web page consumes almost all of the 256-Kbps circuit bandwidth Largest transaction is a download of a 352 KB graphic This single graphic takes 18 seconds to download test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Performance Predictive Studies The use cases for predictive studies: • Predict application pp behavior under different network conditions – Problematic applications • Evaluate performance impact of application design decisions – Modified applications – Problematic application • Validate infrastructure readiness for new deployments test 81 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 40 2/10/2011 Prediction in AppTransaction Xpert • What can predictive analysis tell you? – Response time of application under different circumstances – Will addingg bandwidth help? p – Is this application sensitive to latency? – Can we roll this out to remote users? • What sort of studies will it help with? – Troubleshooting performance problems – Testing a new application – Deploying a current application to new users test – Server relocation studies 82 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Application Deployment Workflow Identify performance issues Capture and import into ACE Create report and prediction scenario templates Use QuickPredict to predict performance test O ti Optional: l U Use M Multi-User lti U QuickPredict for basic capacity planning Optional: Use IT Guru for network and server capacity planning 83 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 41 2/10/2011 Application Deployment Workflow • Optional: Use simulation for further studies – Discrete Event Simulation to predict response times or perform server capacity planning – Flow Analysis for network capacity planning – Simulation covered later in course test 84 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. QuickPredict test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 42 2/10/2011 QuickPredict • Simplified simulation engine • Helps study effects of the network on the application •Q Questions Q QuickPredict can answer – How much bandwidth would it take to get a 4 second response time? – Would increasing the TCP Window improve response time? – What happens to response time if latency increases? – How sensitive is this application to packet loss? – What would the response time be if the link was 65% utilized by other traffic? – What h if I had h d a 10% WAN A Acceleration A l i in i my Networkk – NEW !! test 86 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. QuickPredict: Outputs • Two different output styles – QuickPredict Bar Charts shows components of delay for different scenarios – QuickPredict plots response time over range of network conditions test 87 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 43 2/10/2011 QuickPredict Bar Charts: Scenarios • Click to add multiple scenarios • Right-click to remove scenarios test 88 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. QuickPredict Bar Charts: Options • Put parameters on slider – Drag slider back and forth to dynamically update bar chart • Rename scenarios • Symmetric Bandwidth – Set asymmetric values for bandwidth • Label delay categories • Load and save parameter settings using templates test 89 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 44 2/10/2011 QuickPredict Bar Charts: Options – NEW Introduce WAN Acceleration • • • • Byte Reduction Turn Reduction C Compression i Caching test 90 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. QuickPredict Bar Charts: Tabular Results • View detailed results for each scenario in the Tabular Results tab test 91 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 45 2/10/2011 QuickPredict: Response Time Graphs • Shows sensitivity of task response time to network characteristics: – Bandwidth – Latencyy – Packet Loss – Link Utilization – TCP Window Size – WAN Acceleration test 92 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. QuickPredict: Advanced View • Choose network path to modify • Choose an X-axis parameter and the range of values over which to plot response times • Select values for each of the other parameters • Click Update Graph • Overlay multiple curves on same graph • Load and save templates test 93 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 46 2/10/2011 QuickPredict: Table View • Modify parameters on all network paths in the task – Red text indicates X-axis parameter – Use Set X-axis to specify p y X-axis p parameter – Load and save templates test 94 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. QuickRecode test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 47 2/10/2011 QuickRecode • Changes high level application characteristics – Shows sensitivity of task response time to application characteristics: • Application turns • Total application bytes sent • Total processing time in tiers – Use with QuickPredict to see how response times are affected test 96 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. QuickRecode: Use Cases • Use Cases – Performance benefits – Changing database access to reduce chattiness – Demonstrate effect of faster app server Hint: Use QuickRecode to prove effect of optimizing application test 97 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 48 2/10/2011 LAB 3 LAB 3 Predictive Analysis test 98 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Lab 3: Predictive Analysis • Overview – Deploying a new application to several remote locations – Application has client querying an Oracle database – Application has been deployed to pilot site (in same city as database) and poor performance is already apparent – Goal is to determine if the application deployment will be successful – If not, what can be done to improve performance? • Objectives – Perform a deployment study using QuickPredict Bar Charts – Use AppTransaction Xpert to drill down into the app and see what’s going on test – Use QuickRecode to test a possible fix • Starts on page 20 in your lab manual 99 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 49 2/10/2011 Lab 3: Conclusion • Always try to determine “goal” or “objective” of any problematic application – Is there an SLA expectation – What is the user expectations – What are other users, departments, experiencing • Additional useful troubleshooting information to determine – Is the problem reproducible or is it transient – Are all transactions of the application behaving the same test 100 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Agenda Capture Methodologies Capture Data Sources - New Capture Manager Options Usingg AppTransaction pp Xpert p to p perform p packet captures p - Capture Types - Lab 4: Capture Process Using AppTransaction Xpert to filter packet captures - Capture Filters Using AppTransaction Xpert to import packet captures - Import Process - Lab L b 55: Identifying Id tif i P Performance f P Problems bl in i Unfiltered U filt d Traces T test 101 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 50 2/10/2011 Capture Methodologies test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Basic Capture Methodology • Define and understand the problem at hand – What is slow? – Is it the login or another specific task? – How often does it occur? • What is a task? – Isolate an “atomic” Transaction – Loading single web page – From when user clicks “Submit” to screen refresh • Plan your capture – Simultaneous or single-sided – Which tiers are involved in the application? • Aim for as “clean” test a packet trace as possible – – – – 103 Keep measurement as simple as possible Filter out unrelated traffic Capture one occurrence of one task at a time Perform capture numerous times to help eliminate anomalies OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 51 2/10/2011 Atomic Transaction Example (Task) • Web Application of a Bank – Load Login Page • Open a Web Browser • Type yp URL of the Bank • “Enter” • Login Page Loads – Login • Type User credentials • “Login” • Home Page shows up – Check Savings Account • Mouse/cursor around • Click on “Savings Savings Account Account” • Savings Accounts Page Loads test – Logout • Call up Logout screen • “Submit” • Logged out page 104 Red arrow = Start Capture Blue arrow = Stop Capture OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Capture Data Sources test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 52 2/10/2011 Capture Data Sources • Packet traces are the primary data inputs • AppTransaction pp Xpert p supports pp a wide variety y of packet p sources – – – – ACE Capture Agents ACE Live ACE Enterprise Management Server Agentless Capture Options – NEW! • F5 Load Balancers • UNIX, Linux platforms • WAN Accelerators test • Cisco NAM – Wireshark and other Winpcap and Libpcap sources • TCPDump, Snoop 106 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Agentless Capture Options – NEW • No software installation required • Use the AppTransaction Xpert Capture Manger to control captures remotely via SSH • Supported on platforms that support TCPDump – TCPDump file remains on machine until capture is completed, then deleted after retrieval • Included in the AppTransaction Xpert Plus license test 107 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 53 2/10/2011 Agentless Capture Options – NEW (cont.) • F5 Load Balancers – Excellent way to follow a transaction to the proper server – Perform successful trace file merging across firewalls – Rule out the device as a source of delay • UNIX-Based Operating Systems – Supported Platforms • Linux • Solaris SPARC • Solaris Intel/AMD • AIX • FreeBSD test • HP-UX Itanium • Macintosh PowerPC • Macintosh Intel • True64 108 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Agentless Capture Options – NEW (cont.) • WAN Acceleration Devices – Support for Riverbed and Cisco – Restore visibility of the WAN – More information can be found in session 1471 - Monitoring and Troubleshooting Applications in a WAN Accelerated Environment with ACE Live™ and AppTransaction Xpert test 109 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 54 2/10/2011 Capture Data Sources – AppResponse Xpert (ACE Live) • New seamless 1-click integration – Let ACE Live do the filtering for you • Easily pull traces from alert snapshots off actual t l application li ti issues i test 110 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Capture Data Sources – AppForensics Xpert (ACE EMS) • ACE Enterprise Management Server – Provides web-based extension to Continuous Capture functionality – Supports both End-User Driven Capture workflows, as well as Server-Driven Capture Workflows test – End Users report application slowness • Trouble ticket is created with trace files of the incident attached • Troubleshooters claim problem reports and troubleshoot on their schedule 111 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 55 2/10/2011 AppForensics Xpert – Web Capture Retrieval test 112 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Capture Data Sources – AppForensics Xpert • Server-Driven Capture Workflow – Panorama provides forensic actions based on threshold violations within the application servers – The trace file forensic action automatically generates a trace that is accessed through AppForensics Xpert – For more detail, attend session 1482 – Deploying an Enterprise-Wide Packet Capture Architecture with ACE EMS and ACE Live test 113 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 56 2/10/2011 Capture Data Sources – ACE Agents • Benefits of using ACE Agents – Unlicensed – Support for major platforms (Windows, HP-UX, AIX, Solaris, Linux, Mac OS X, FreeBSD) – Lightweight, Secure, and Stable – Silent Install capability – Server Statistics collection (Windows) – Enhanced Filtering (NEW) – On-Demand and Continuous Capture modes test 114 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Additional Data Sources • AppTransaction Xpert can import a variety of additional data – Identify these sources during capture to ensure collection • From OPNET products – Windows ACE Capture Agents can collect Lightweight Server Statistics – OPNET Panorama • Server statistics and performance metrics • Java transaction tracing • Third-party sources – HP GlancePlus – Perfmon – Custom datatest in comma saved (.csv) format 115 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 57 2/10/2011 Capture Lightweight Server Statistics • Capture Windows performance data – ACE Capture Agent generates two files – Application traffic (*.appcapture) file – Windows performance data (*.appcapture.da) file Display internal server metrics • Percent Processor Time • File Read Bytes Per Second • System Calls Per Second • Available MB • File Write BytestestPer Second 116 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. AppInternals Xpert (Panorama) Server Statistics • Capture Server Statistics from AppInternals Xpert – Use with on-demand captures – Thousands of available metrics • For more information: – Session 1461- OPNET Panorama® Essentials: Multi-Tier Application Monitoring, Correlation, and Troubleshooting test 117 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 58 2/10/2011 AppInternals Xpert Transaction Trace • AppInternals Xpert can create a trace of all activity through the Java Virtual Machine (JVM) or .NET CLR – Provides a call-by-call sequencing (or call stack) of all activity within your application li i • This transaction trace can be imported into AppTransaction Xpert to explain what is occurring on a given system during long processing delays test 118 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Capture Manager • Capture Manager control parameters test 119 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 59 2/10/2011 Capture Agent Settings test 120 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Capture Types test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 60 2/10/2011 On-Demand Capture • Methodology –User knows exactly when a problem is going to occur and what it takes to reproduce it –Deploy and configure ACE Capture Agents –Start the capture –User re-creates the problem –Stop the capture Import capture for analysis –Import test 122 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. On-Demand Capture (cont.) test 123 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 61 2/10/2011 On Demand Capture Process: Adding an Agent test 124 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. On Demand Capture Process: Specify Agent Settings test 125 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 62 2/10/2011 Capture Process: Start Capture test 126 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Capture Process: Finish Capture test 127 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 63 2/10/2011 Capture Process: Open in Transaction Analyzer test 128 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Scheduled Capture • Useful for reproducible problems during off hours • Methodology –User knows when a problem will occur • For example, after-hours batch process or database backup –Schedule a capture • Choose one-time or repeating capture • Set the schedule for a capture • Specify schedule for download –Import I t trace t for f analysis l i test 129 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 64 2/10/2011 Scheduled Capture (cont.) test 130 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Scheduled Capture (cont.) Scheduled capture can create one-time, daily, weekly, or monthly capture schedules test Scheduled capture utility can optionally create a one-time, daily, weekly, or monthly schedule to download trace files 131 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 65 2/10/2011 Continuous Capture • Useful for troubleshooting intermittent problems • Methodology –User experiences intermittent problem –Start continuous capture –User calls help desk • Problem occurred again around 9:30 –Preview capture –Download Download trace of problematic time span test –Import trace for analysis • Agent details –Same capture agents as on-demand capture 132 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Continuous Capture (cont.) test 133 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 66 2/10/2011 Continuous Capture: Preview/Download test 134 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. LAB 4 LAB 4 Capture Process test 135 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 67 2/10/2011 Lab 4: Capture Process • Overview – Configure a capture agent – Perform an on-demand capture p – Open in Transaction Analyzer • Skills covered – Learn how to configure a capture agent – Understand how to use agent filters – Capturing from the network test • Starts on page 32 in your lab manual 136 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Lab 4: Conclusion • Configured capture agent for capture – Located and specified host IP Address – Specified capture mode – Set up a filter • Performed a capture – Started the agent – Performed a transaction – Stopped the agent – Brought g the capture p file into Transaction Analyzer y test 137 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 68 2/10/2011 Capture Filters test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Filtering Unrelated Traffic • You want to capture the transaction of interest – AppTransaction Xpert lets you filter out irrelevant traffic • Options to filter during capture, during the import or after the import – The earlier the better • Why filter during capture? – Only capture the specific transaction – Smaller trace file to process => easier/faster import • Why not filter during capture? – Sometimes you want to see the whole picture – If you’re not 100% sure about application path • Recommended test methodology – Capture without filters for short duration – Examine trace – Determine best filter to use – Capture subsequent tasks with filter 139 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 69 2/10/2011 Filter: During Capture Users can filter out or only allow traffic based on a specific protocol Users can filter specific hosts and/or ports for incoming or outgoing traffic New Filtering Options in Capture Agent test 3.9 140 Use wildcards and slash notation Use DNS names Port ranges Broadcast and multicast can be excluded independently Users can elect to exclude or allow broadcast and multicast traffic OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Filter: During Capture (cont.) • Preview in Continuous Capture – Select the IP addresses, conversations and time ranges of interest • Great for troubleshooting intermittent issues test 141 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 70 2/10/2011 Filter: During Import • What is Trace Explorer? – Graphical protocol analyzer – Used to helpp filter unrelated traffic during import process • Why would you use it? – Graphically filter a capture file – Allows for fast read of large trace files – You want to know what you’re looking at test 142 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Filter: After Import • Once capture file has been imported, users can still filter irrelevant traffic – Right-click on any tier pair conversation, application message, application li i transaction i or networkk packet • Option to Exclude or Permanently Delete – Filter Wizard • Remove unidirectional, non-IP, UDP, broadcast traffic – Capture File Filter • Use to remove traffic by host, port, protocol 143 test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 71 2/10/2011 Import Process test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Import Overview • Now that the data has been captured, it must be imported into AppTransaction Xpert • Review what has been captured – Do you have a single capture file? – Do you have more than one capture file? – Are you capturing WAN accelerated traffic? – Is the capture file filtered? • If not, use Trace Explorer • Greatly simplified for 16.0 !! test 145 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 72 2/10/2011 Import Changes in 16.0 • Import process has been greatly simplified – 15.0 required answering a series of questions about tier locations, bandwidths, and latencies – For 16.0, we focused on asking as few questions as possible • Filter, view statistics or diagnoses, or explore your transaction immediately • Locations, bandwidths, and latencies can still be specified to break down Network Transfer effects into Bandwidth, Protocol, and Congestion • Most importantly, you can change these values whenever you need to • Import is now broken down into use cases – Single packet trace – Merge several traces – Merge traces from a WAN-accelerated environment test – Batch import – Filter into Trace Explorer 146 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Import a Single Capture File test 147 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 73 2/10/2011 Merging Capture Files • What is a merge? – Two or more traces of an application transaction can be combined – Creates unified view of application – Accurately determines the send and receive times of each packet • Benefits and use cases – Allows analysis of multi-tier transactions where all traffic does not go through one location – Improves analysis in a production environment with congestion issues such as packet loss and retransmissions test 148 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Merging Capture Files (cont.) • All traces must be of the same transaction • All must have been captured at the “same time” • Easier to import p if there is some common traffic between traces test 149 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 74 2/10/2011 Merging Capture Files (cont.) File name test Load previously saved import details (tier names, tier locations, bandwidth, latency,…) 150 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Merging Capture Files (cont.) • Replace IP addresses with friendly names (can use DNS) test 151 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 75 2/10/2011 After Opening/Merging • Now what? – Filter down to a single transaction or remove user think time – Open AppDoctor • Note that there are only Processing and Network effects identified – Specify bandwidth and latency to break out individual network effects test 152 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. AppDoctor Bandwidth, protocol and congestion are all included in one category named “Network Transfer” test 153 To break down the various components t off “Network “N t k Transfer”, click the “Refine Network Effects…” button OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 76 2/10/2011 Refine Network Effects Specify the local and remote bandwidths here test 154 Tier locations and latencies are automatically detected, but can be changed if needed OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Specify Bandwidth and Latency • Minimum bandwidth – Capacity of the slowest link – Should be a known factor about your network • Minimum one-way latency – – – – PathProbe provides an estimate Half the value returned by ping Auto-calculated in a trace merge In manyy cases,, AppTransaction pp Xpert p can estimate latencyy of a single-sided g trace test 155 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 77 2/10/2011 Specify Bandwidth and Latency (cont.) • Quiz – Bandwidth of each segment is known – Ping reports 46ms RTT test • What would you specify for bandwidth and latency between the Client and the Server during import into AppTransaction Xpert ? Bandwidth = 1544 Kbps 156 Latency = 23ms OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. AppDoctor – After Network Definition test BEFORE 157 AFTER OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 78 2/10/2011 LAB 5 LAB 5 Identifying Performance Problems in Unfiltered Traces test 158 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Lab 5: Identifying Performance Problems in Unfiltered Traces • Overview – Open a single side capture – Filter traffic – Refine network effects – Analyze transaction • Skills covered – Step through the import process in AppTransaction Xpert – Understanding the importance of bandwidth settings test • Starts on page 42 in your lab manual 159 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 79 2/10/2011 Lab 5: Conclusion • Imported a single-sided capture file – Imported capture file – Filtered irrelevant traffic – Estimated locations – Entered local and remote bandwidth – Estimated latency test 160 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Introduction to AppTransaction Xpert test Day 2 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 80 2/10/2011 What to Expect in Advanced AppTransaction Xpert Topics • Mix of lecture and lab • Advanced class – Address network concepts important to using AppTransaction Xpert – Dive into “the science” behind ACE – Highlight new ACE features – Discuss TCP • Please be interactive – Participate test – Ask questions 162 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Prerequisites • Prerequisites – Reviewed yesterdays material – Reviewed session 1441 – Equivalent experience • You should already understand the mechanics of ACE – Capturing a trace – Importing traces – Generating reports • You should already recognize “vanilla” problems test – Chatty applications – Insufficient bandwidth – Slow servers –… 163 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 81 2/10/2011 Agenda How AppTransaction Xpert Calculates and Predicts Components of Delay - Lab 1: Calculating Network Delays Understanding the different Components of Network Delay - Congestion Delay - Protocol Delay - Lab 2: Understanding Components of Delay Understanding Parallel Effects - Parallel Effects - User Think Time test - Lab 3: Understanding Parallel Effects 164 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. How AppTransaction Xpert calculates and predicts p of Delayy components test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 82 2/10/2011 Components of Application Delay • The following are key components of delay in a transaction: – Processing • Client • Server(s) – Network • Protocol (TCP WIN, Nagle’s, etc.) • Bandwidth • Latency (baseline, dominated by network hardware and ultimately ruled by C) • Congestion (including queuing) • Packet loss – Application itself test • Think of the top two topics as “resources”. How does the app use them? – Efficient/inefficient code – App turns/Chattiness 166 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Review of Network Delays • Packets experience a complex series of – Network hops – Technologies – Queuing delays –… T1 OC3 test 167 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 83 2/10/2011 Apps See the Network as a “Black Box”, So Does AppTransaction Xpert • For most troubleshooting, complexity can be simplified • Think of entire network as a single “equivalent” circuit • Once packet leaves source, the delay can be separated into 3 categories – Bandwidth delay – Latency delay – Congestion (queuing) delay • What about Protocol? – Protocol delay is a function of the sender/receiver T1 OC3 – Packet Loss is handled by the protocol test Bandwidth, Latency, Congestion 168 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Bandwidth Delay • Time it takes to stream all bits onto circuit • This number varies with packet size • Can be calculated from ppacket size and circuit bandwidth 2000 bits Headers Data 2000 bits/second …01110010010 01110010010 test What is the bandwidth delay for this packet? 169 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 84 2/10/2011 Calculate Serialization Delay How long does it take to serialize this packet on to the wire? Serialization and BW are the same thing • If you know the bottleneck bandwidth across a link and the size of each test packet, you know how long it takes to get across the link due to serialization delay (ie: Bandwidth delay) • Note: we are ignoring latency at this point 170 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Latency Delay • Time it takes for signal (1st bit) to travel across network – Distance / speed of light – Normal delays caused by device forwarding • This is a constant number for all packets 7 ms 1st bit 10 ms 1 ms 10 ms 1 ms 10 ms 1 ms What is the latency for this circuit? test What value would “ping” produce? 171 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 85 2/10/2011 Application Turns • A turn is a change in the flow direction of application data • Turns magnify circuit latency • An application pp with no turns would onlyy experience p latency y once • An application with 99 turns would experience latency 100 times • Simple equation for total latency delay – Latency Delay = circuit latency * (turns +1) – This equation only works on sequential applications No Application pp Turns Manyy Application pp Turns test 172 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Congestion Delay • Queuing on devices • Usually caused by other traffic • Can varyy from packet p to packet p Forwarding rate = 1 packet / ms packet test 5 packets in forwarding queue What is the congestion delay for this device? What is the total delay across this device? 173 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 86 2/10/2011 Determine Queuing/Congestion Packet should have taken thi time this ti . . . . . . but ended up taking this time test • If a packet should take x time but takes y time, y – x yields queuing/congestion delay 174 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Example Network Delay • Bandwidth = 1,000 bps • Latency = 5 seconds T=0 10 20 30 40 50 Client Bandwidth delay 30 seconds 30,000 bits Latency delay 5 seconds Congestion delay test Server 15 seconds 50 175 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 87 2/10/2011 Packet Trains • Stream of packets that are sent together • Arrive at destination spread out • Manyy TCP segments g of a larger g application pp message g • Spreading caused by limited bandwidth of bottleneck link T=0 10 20 30 40 50 60 10 20 30 40 50 60 test 0 176 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Calculating Delay for Packet Trains • Treat packet trains like 1 big packet • Same rules apply – First—establish delay caused by bandwidth (stream bits onto wire) – Second—add latency (last bit of the packet train getting across) – Third—anything left over is congestion Application Layer Transport Layer test Network Layer 177 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 88 2/10/2011 Calculating Delays—Advanced • Bandwidth = 1,000 bps • Latency = 5 seconds T=0 10 20 30 40 50 60 70 80 90 100 Client Bandwidth Delay 5 + 10 + 30 Latency Delay 5 +5 +5 Congestion Delay 5 + 15 Tier Processing (Client) 10 test 10 5,000 bits 20 Server 65 80 Tier Processing (Server) 10 100 10,000 bits Total Delay = 45+15+20+10+10 = 100 178 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Pie Chart • AppTransaction Xpert creates a pie chart that shows all of the components of delay in one screen • Shows the total impact on response time due to each individual effect • Each wedge is the maximum benefit that a single upgrade will provide Bandwidth 45 Bandwidth Latency 15 Latency Congestion 20 Client 10 Server 10 Congestion Client Server test 179 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 89 2/10/2011 Network Effects in App Doctor • AppTransaction Xpert 16 does not require fully specifying network data for import • Creates a “Network Effects” pie slice test 180 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Network Effects • Specifying Bandwidth will cause AppTransaction Xpert to recalculate the values for the components of delay test • For more information, see Session 1442: Application Capture and Import Strategies with AppTransaction Xpert 181 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 90 2/10/2011 Calculating Delays • Two ways to calculate the Pie Chart – Adding up (add the components of delay) – Squeezing down (reduce effects one at a time) T=0 10 20 30 40 50 60 70 80 90 100 Client test Server 10 182 20 65 80 100 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Review of Network Delays • Different factors can affect network delay – Bandwidth – Latency – Congestion • Some factors are constant and some are variable • Network delay is not always due to lack of bandwidth • Important that you understand fundamentals to interpret ACE results test 183 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 91 2/10/2011 LAB 1 LAB 1 Calculating Network Delays test 184 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Lab 1: Calculating Network Delays • Overview – Apply knowledge of sources of delay to a real application • Objectives – From the raw statistics calculate total bandwidth delay – From the raw statistics calculate total latency delay – Compare results with ACE analysis • Starts on page 64 in your lab manual test 185 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 92 2/10/2011 Lab 1: Conclusion Response Time = 26.03 Network Data = 3,199,760 Bytes Size of link = 1,544,000 Kbps p Bandwidth Delay = (3,199,760*8)/1,544,000 = 16.58 Percentage of Bandwidth = 16.58/26.03 = 63.7% Turns = 337 Latency = 24.23ms Latency Delay = 24.23*(337+1) = 8.18 seconds Percentage P t off L Latency t = 8.18/26.03= 8 18/26 03 31.4% 31 4% test 186 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Lab 1: Conclusion Calculated that Bandwidth caused 63.7% of response time Calculated that Latency caused 29.8% of response time test 187 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 93 2/10/2011 Lab 1: Conclusion Why didn’t the math work out perfectly? Some amount of congestion in trace Some amount of protocol delay in trace Equation E ti didn’t did ’t accountt for f bi-directional bi di ti l traffic t ffi Our simple math was very close Important to perform this type of “sanity check” test 188 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Congestion Delay test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 94 2/10/2011 Congestion Delay It is somewhat amazing that congestion can be calculated Very difficult to measure and calculate manually ACE can measure the effect even without seeing the cause ACE provides 2 important pieces of information for congestion Actual congestion delay for each packet The cumulative effect of congestion on the response time test 190 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Key Concept Trace file includes 1 piece of time information Time the packet arrived at the capture probe 1 piece of time information per packet ACE includes 2 pieces of time information for each packet Time the packet left the sending side Time the packet arrived at the receiving side test 2 pieces of time information per packet 191 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 95 2/10/2011 Key Concept ACE knows both the send and receive time Delay analysis requires that both values are known How does ACE derive the second ppiece of time information? Trace Merge Single Trace Adjust test 192 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Trace Merge Each trace shows what happened locally Once clocks are aligned, draw the packets from send to receive Trace captured at client side T = 0 1 2 3 4 5 6 7 8 9 10 Client T = 0 1 2 3 4 5 6 7 8 9 10 Client A BC C* A B C C* D D Trace captured at server test side AB C* D A B Server C* D Server 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 193 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 96 2/10/2011 Single Side Adjust With a single trace, ACE only knows what happened locally Must estimate what happened on network and at other side Based on bandwidth and latency values set during import ACE estimates ti t when h th the packet k t was sentt ACE estimates when the packet will arrive T = 0 1 2 3 4 5 6 7 8 9 10 Client A B C C* E Trace captured at client side T = 0 1 2 3 4 5 6 7 8 9 10 Client A BC test C* E Server 0 1 2 3 4 5 6 7 8 9 10 194 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Trace Merge What happens if the import parameters are incorrect Bandwidth is set too low g Latencyy is set too high Might create false packet crossings This will confuse analysis T = 0 1 2 3 4 5 6 7 8 9 10 Client A B C D E Trace captured at client side T = 0 1 2 3 4 5 6 7 8 9 10 Client A BC D test E Server 0 1 2 3 4 5 6 7 8 9 10 195 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 97 2/10/2011 Key Concept With BOTH send and receive time ACE knows Actual packet delay packets were sent and received Order of how p What would happen if we graphed the delays? testTotal 196 Packet Delay OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Packet Size vs. Delay Graph Delay (sec) • Graph the packet size vs. total packet delay Congestion Slope is bandwidth* test Pure latency Packet Size (bits) * Inverse of bandwidth (1/BW) 197 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 98 2/10/2011 Packet Size vs. Delay Graph (cont.) Delay (sec) • Graph Packet Size vs. Total Packet Delay test Packet Size (bits) 198 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Viewing Congestion in ACE • Several good ways to view congestion in ACE – AppDoctor diagnosis – Summary of Delays – Congestion drill-down – Data Exchange Chart test 199 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 99 2/10/2011 Congestion in the AppDoctor Diagnosis • Specifically marks category as a Bottleneck test 200 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Congestion in the Summary of Delays • Shows the cumulative effect of the congestion on response time • How much slower was the application because of congestion? test 201 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 100 2/10/2011 Drilling into Congestion • Drilling into congestion can show if the packets are getting queued or test otherwise slowed down 202 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Drilling into Congestion (cont.) • Look for “plateaus” test • Always keep in mind the scale of the chart! • Is delay uniform or random? – Is there any encryption on the link? 203 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 101 2/10/2011 Congestion in the Drill-Down • Packet Congestion graph • Retransmission graph test 204 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Congestion in the DEC • Network chart layered with Packet Congestion graph test 205 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 102 2/10/2011 Congestion Review • Congestion Delay is cause by other traffic on the network • Always better to merge traces from congested circuits – Congestion values for a trace merge are precise – Congestion values for a single-side adjust are approximations • Several ways to see congestion and its effect on the application • Imagine “squeezing” the congestion out of the trace to calculate its contribution to overall delay test 206 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Protocol Delay test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 103 2/10/2011 Protocol Delay • Caused by the artificial throttling of traffic • Packets are prevented from even getting onto the network • Caused mainlyy byy TCP test 208 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Why Does TCP Exist? • TCP offers reliable data transport to applications – Application data is guaranteed to be delivered – Packets that are “lost” by the network will be retransmitted • TCP makes applications behave in a network-friendly manner – Throttles the rate at which application sends data – Ensures that no single user consumes all of the bandwidth • Both can impact application performance test 209 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 104 2/10/2011 What Does Protocol Delay Look Like? • Protocol delay prevents packets from even getting on the network Application Layer Transport Layer Network Layer No Protocol Delay Protocol Delay test Protocol Delay 0 1 2 3 4 5 6 7 210 0 1 2 3 4 5 6 7 8 9 10 11 12 13 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Observe Protocol Behavior Protocol delayy is when the sending side delays or throttles sending of data (this is different than processing delay) • Observe how data leaves in stages indicating Protocol Delay test • Observe other attributes like TCP window size • Our example here is ridiculously simplified, but real protocol delay is observable in a real trace 211 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 105 2/10/2011 Forms of Protocol Delay • Critical that you understand mechanisms of TCP • Several different TCP mechanisms cause can protocol delay – TCP windowing – Slow Start – Frozen window – Nagle’s algorithm – Retransmissions – Out of sequence packets test 212 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Viewing Protocol Delay in ACE • Several ways to view Protocol Effects –AppDoctor diagnosis –Summary S off D Delays l –Protocol drill-down –TCP Visualization in DEC test 213 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 106 2/10/2011 Protocol Delay in the AppDoctor Diagnosis • Specifically marks category as a Bottleneck test 214 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Protocol Delay in the Summary of Delays • Double-click on slice to get more detail test 215 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 107 2/10/2011 Protocol Delay Drill-Down • Graphs dynamically appear based on AppDoctor Diagnosis • Users can manually add or remove graphs • Protocol Delayy – TCP In-Flight Data – Retransmissions – Out of Sequence Packets – TCP Nagle’s Delay – TCP Advertised Receive Window test 216 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Protocol Delay Review • Protocol delay results from the artificial throttling of traffic • Several different TCP mechanisms can cause it • Prevents the ppacket from even getting g g onto the network • Can often be calculated without a trace merge • Imagine “squeezing” the protocol effects out of the trace to calculate its contribution to overall delay test 217 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 108 2/10/2011 LAB 2 LAB 2 Understanding Components of Delay test 218 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Lab 2: Understanding Components of Delay • Overview – User is transferring files with Microsoft file browser • Transfer time is very slow • Want to understand cause of delay • Objectives – Identify and understand the sources of delay – Compare slow transfer to fast transfer • Starts on page 69 in your lab manual • Homework test – Continue with homework sections of lab manual if you have time 219 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 109 2/10/2011 Lab 2: Conclusion • File transfer is slow • Primary source of delay is protocol/congestion • Some “potential” problems with TCP but no smoking gun • Look at packet delay graph to evaluate congestion • Significant congestion on WAN circuit – Base latency is 25ms – Some packets take 125 ms to cross the network test 220 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Lab 2: Conclusion • Congestion is primary cause of delay • Removing other traffic on network fixes problem – Reduces congestion – Response time decreases • Might be a case for QoS Significant Congestion No Congestion test 221 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 110 2/10/2011 Parallel Effects test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. The “Mysterious” Gray Wedge • AppTransaction Xpert shows “parallel effects” as a gray pie wedge – Transaction has more than one effect taking place at one time test – HTTP is an example of a transaction that exhibits parallelism 223 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 111 2/10/2011 Parallel Effects Drill-Down • Can show the individual effects with their combined parallel effects together • Solid bars are the individual effects • Striped p bars are the parallel p effects • Optimizing both components simultaneously will give the benefits of the individual effects AND the parallel effects T=0 10 20 30 40 50 60 70 80 Bandwidth 60% Latency 10% Server 20% Parallel 10% Response Time 90 100 Parallel Effects of Bandwidth and Latency test Parallel Effects of Server and Latency 224 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Understanding Parallel Effects • Parallel delays may seem counterintuitive at first • It is a real effect and does happen in the real world • Real Life Parallel Effects example p that yyou will remember! – Reading a paper while eating breakfast – Cooking a meal (dessert in the oven while steak is on the grill) • Would be difficult to split these into simple 2 category pie charts – Incorrect to say grilling steak was 50% and dessert was 50% ? – Suggests that eliminating dessert would cut dinner preparation in half – Probably more like 25% steak, 25% dessert, 50% both – Life is parallel test 225 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 112 2/10/2011 Types of Applications • Applications fall into two main categories –Transactional • Strong St cause and d effect ff t relationship l ti hi bbetween t packets k t • Each application message is dependent on the one before it • Examples – Database queries – Web transactions – File transfers –Asynchronous • No strong cause and effect relation between packets test • Examples – Voice – Citrix 226 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Types of Applications • Transactional application can be further divided –Sequential • Single Si l execution ti path th • Application is only doing one thing at a time –Parallel • Two or more simultaneous execution paths • Application is doing multiple things at a time test 227 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 113 2/10/2011 Types of Applications • Sequential (Adding up method) – Each slice of time can be placed into a single delay category – The components of delay add up to the total response time T=0 10 20 30 40 50 60 70 80 90 100 Client test Server 228 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Types of Applications • Parallel (Squeezing down method) – Some slices of time could be placed in multiple delay categories – The components of delay would up to more than the total response time T=0 10 20 30 40 50 60 70 80 90 100 Client test Server Connection 1 Connection 2 Connection 3 229 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 114 2/10/2011 AppDoctor Individual Effects • Summary of Delays for a sequential transaction Response Time 6 sec Server 2 sec (33%) Client 2 sec ((33%)) Bandwidth 2 sec (33%) test 230 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. AppDoctor Parallel Effects • Summary of Delays for a parallel transaction • Server is now processing while simultaneously sending the large message test 231 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 115 2/10/2011 AppDoctor Parallel Effects (cont.) How does AppDoctor handle multiple effects occurring at the same time? test The "Parallel Effects" category Response Time 6 sec Server 2 sec (33%) Client 2 sec (33%) Parallel Effects 2 sec (33%) Mixed wedge shows total impact due to server processing and bandwidth occurring in parallel Because there may be many mixed wedges, ACE shows these as gray in the Pie Chart 232 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Understanding Parallel Applications • Calculating delays for sequential transactions is fairly easy – Each slice of time can be attributed to a single cause – All the slices add up to the total • Calculating delays for parallel transactions can be complex – A slice of time may be attributed to multiple sources – Only the squeezing method works here • Parallel Effects are subtle and not always easy to visualize • Parallel Effects do not always occur just because delays overlap • Aren’t you glad you have AppTransaction Xpert to do this for you test 233 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 116 2/10/2011 Parallel Effects Example • Parallel – Two independent connections occur in parallel – Both must complete for the application to complete Client Conn. B B B B A Conn. A test Server 234 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Understanding Parallel Effects • Slide Conventions – Show independent connections as parallel DECs – Keep in mind that these are happening on top of each other – Obvious now why you can’t just add delays up T=0 10 20 30 40 50 60 70 80 90 100 Client Connection A Server T=0 10 20 test 30 40 50 60 70 80 90 100 Client Connection B Server 235 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 117 2/10/2011 Understanding Parallel Effects • Parallel applications with squeezing method – Perform experiments that squeeze out each category of delay • Color associated with that delay would disappear • Response time of transaction may decrease – Record the difference from each experiment T=0 10 20 30 40 50 60 70 80 90 100 Client Server T=0 10 20 test 30 40 50 60 70 80 90 100 Client Server 236 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Understanding Parallel Effects • What would happen if bandwidth became infinite? – Green would be squeezed out – Total response time of the transaction would change to 40 seconds – Bandwidth is responsible for 60 seconds of response time (60%) T=0 10 20 30 40 50 60 70 80 90 100 Client Server T=0 10 20 test 30 40 50 60 70 80 90 100 Client Server 237 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 118 2/10/2011 Understanding Parallel Effects • What would happen if latency became zero? – Red would be squeezed out – Total response time of the transaction would change to 90 seconds – Latency is responsible for 10 seconds of response time (10%) T=0 10 20 30 40 50 60 70 80 90 100 Client Server T=0 10 20 test 30 40 50 60 70 80 90 100 Client Server 238 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Understanding Parallel Effects • What would happen if the server became infinitely fast? – Yellow would be squeezed out – Total response time of the transaction would change to 80 seconds – The server is responsible for 20 seconds of response time (20%) T=0 10 20 30 40 50 60 70 80 90 100 Client Server T=0 10 20 test 30 40 50 60 70 80 90 100 Client Server 239 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 119 2/10/2011 Understanding Parallel Effects • Something interesting just happened – Bandwidth caused 60%, Latency caused 10%, Server caused 20% – The individual sources of delay are only responsible for 90% – What caused the remaining 10%? • Delays that are not removed by a single source are Parallel Effects Bandwidth 60% Bandw idth Latency test 240 Server Parallel Latency 10% Server 20% Parallel 10% OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Understanding Parallel Effects • Squeeze out 2 effects at the same time – Bandwidth = 60% – Latency = 10% – Server = 20% – Parallel Effects = 10% T=0 10 20 30 40 50 60 70 80 90 100 Client Thread A Server T=0 10 20 test 30 40 50 60 70 80 90 100 Client Thread B Server 241 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 120 2/10/2011 Understanding Parallel Effects • Remove the Server and Latency together – Server by itself is 20%, Latency by itself is 10% – The pair is responsible for 35 seconds of response time (35%) – Parallel interaction of the 2 provides a bonus benefit if changed simultaneously T=0 10 20 30 40 50 60 70 80 90 100 Client Thread A Server T=0 10 20 test 30 40 50 60 70 80 90 100 Client Thread B Server 242 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Understanding Parallel Effects • Remove Bandwidth and Latency together – Bandwidth by itself is 60%, Latency by itself is 10% – The pair is responsible for 75 seconds of response time (75%) – Parallel interaction of the 2 provides a bonus benefit if changed simultaneously T=0 10 20 30 40 50 60 70 80 90 100 Client Thread A Server T=0 10 20 test 30 40 50 60 70 80 90 100 Client Thread B Server 243 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 121 2/10/2011 Parallel Effects Drill-Down • • • • Can show the individual effects with their combined parallel effects together Solid bars are the individual effects Striped bars are the parallel effects Optimizing both components simultaneously will give the benefits of the individual effects AND the parallel effects T=0 10 20 30 40 50 60 70 Parallel Effects of Bandwidth and Latency 80 90 60% Latency 10% Server 20% Parallel 10% 100 Response Time Parallel Effects of Server test and Latency 244 Bandwidth OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Parallel Effects Summary • Delays that can not be removed by a simple change of one parameter • To reduce, you must change at least two parameters simultaneously • These delays y are not bad – Application is efficient – Application is doing multiple things at the same time test Bandw idth Latency Server Parallel 245 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 122 2/10/2011 User Think Time test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Capturing a Good Transaction (Task) • Important to get an “atomic” transaction – Atomic transaction is the smallest transaction that you can capture effectively – Capture should be from edge of user input to screen refresh – Eliminate “PEBCAK” test 247 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 123 2/10/2011 Atomic Transaction Example • Modify Customer Record – Login • Call up the Login screen • Type information • “Submit” – Look up record • Call up correct screen/menu • Type information • “Submit” – Modify data • Mouse/cursor around • Type information test • “Submit” – Logout • Call up Logout screen • “Submit” 248 • End user complains “modifying a customer record is slow” • Need to find out what is involved in transaction Red arrows indicate the key part of the transaction to capture OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Identifying and Removing User Think Time • Always get as close as you can to a perfect “Atomic” transaction as described in the Capture p Section • However, sometimes you just can’t get the perfect capture • “User think time” enters the trace • AppTransaction Xpert deals with think time – Auto detects when calling up AppDoctor – Allows user to explicitly specify think time test 249 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 124 2/10/2011 Removing User Think Time AppTransaction Xpert automatically identifies that the trace contains user think time. This dialog comes up when AppDoctor is selected AppTransaction Xpert also allows the user to test pick any dependency on the client (first talker) and categorize it as user think time 250 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Removing User Think Time Original Analysis test User Think Time Removed 251 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 125 2/10/2011 LAB 3 LAB 3 Understanding Parallel Effects test 252 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Lab 3: Understanding Parallel Effects • Objectives – Explore an application transaction that has significant parallel effects • Trace is fairly simple • Trace is somewhat artificial – Practice visualizing the squeezing method to understand the behavior – Use Drill-down and QuickPredict to understand the interactions • Starts on page 84 in your lab manual test 253 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 126 2/10/2011 Lab 3: Conclusion • Parallel delays may seem counterintuitive at first • It is a real effect and does happen in the real world • This example p is exaggerated gg to make it easyy to visualize test 254 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Agenda Overview of TCP - TCP Basics - TCP Windowing - Lab 4: Examining TCP Behavior Troubleshooting T bl h ti TCP - Multi-User QuickPredict - Bandwidth-Delay Product - Lab 5: Troubleshooting a Slow File Transfer - Detecting Packet Loss - Frozen Window - Nagle Nagle’ss Algorithm test Discrete Event Simulation Instructor Demo: Nagle’s Algorithm using DES References - What’s New ? - Wrap up 255 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 127 2/10/2011 TCP Basics test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. What is TCP? • Transmission Control Protocol • Transport p layer y protocol p Application Application • Layered between IP and application layer TCP • Typical applications such as – HTTP (Web) – POP, IMAP, SMTP (e-mail) – Telnet, FTP, and more test 257 IP Data Link OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 128 2/10/2011 Why Does TCP Exist? 1) TCP offers reliable data transport to applications • User data is guaranteed to be delivered 2) TCP makes applications behave in a network-friendly manner • TCP may throttle the rate at which it sends user data Both impact application performance test 258 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Which Applications Don’t Use TCP? • TCP adds overhead • TCP limits throughput • Not all applications pp need what TCP provides p – VoIP – Voice over IP – Video Conferencing – SNMP – Simple Network Management Protocol – Network Gaming Protocols – Domain Name System (DNS) – Network Time Protocol (NTP) test 259 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 129 2/10/2011 Motivation – Why Is TCP Important? Example TCP/IP Stack Application Developers Network Managers SOAP SOAP HTTP HTTP SSL SSL TCP TCP IP IP IP ETHERNET FIBER ETHERNET test • TCP is in a key location of networked applications’ protocol stacks – Frequently misunderstood – This makes people and software that understand TCP valuable 260 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Review: ACE Application Message Chart Timeline Tiers Application message arrows Application message groups g p test Packet sizes Red packets have no payload (usually acknowledgements, SYNs, or FINs) 261 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 130 2/10/2011 How to Read TCP Decodes Special flags Size is size of entire packet (including IP, MAC, etc.) Sequence number Receive window Destination port Frame number F b ttaken k ffrom trace file(s). Maytest not always be in strict order due to trace merge Source port Payload length Acknowledgement number 262 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Opening a Connection • Three-way handshake to open connection • Used to exchange – Initial SEQ numbers – TCP Options Cli Client 3 C 1 A test 2 B Server 263 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 131 2/10/2011 Closing a Connection • Four-way handshake to close connection • “I have no more data to send.” • Typical example connection closed at client client 273 B 274 C test 275 D 272 A connection closed at server 264 server OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Closing a Connection (cont.) • Often only three segments used • Second FIN also is an acknowledgement for first FIN (B&C in one packet) • TCP Resets are an alternate way to terminate a connection 86 A 88 D 87 B&C test 265 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 132 2/10/2011 SEQ and ACK Numbers • Ensures that data was received • Sending TCP gives each segment a sequence number – SEQ of segment K is SEQ of segment K-1 plus length of segment K-1 • Receiving TCP acknowledges reception by replying with ACK – ACK number is the SEQ number of the next expected byte • SEQ and ACK numbers are in bytes, not packets Receiver Sender SEQ=100 Q LEN=10 K-1 K test ACK= 110 ? Seq # K-1 266 Seq # K SEQ = 110 ? LEN=55 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. SEQ and ACK Quiz • Which frame(s) is 36 ACKing? • Which frame(s) is 44 ACKing? test 267 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 133 2/10/2011 TCP Acknowledgements • Receiver normally ACKs every other packet – Inefficient to ACK every packet • What if only one packet is received? Data test 268 Acknowledgeement – Delayed-ACK timer must expire – 100-200 ms (OS determines exact delay) – Usually does not impact total application performance OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Review: TCP Basics • TCP is a connection-based protocol • It uses a three-way handshake for opening connections • Each side closes the connection separately p y • SEQ and ACK numbers ensure reliable delivery test 269 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 134 2/10/2011 TCP Windowing test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. TCP Windowing • TCP window size is the maximum amount of data TCP will allow outstanding (“in-flight”) • If window size is less than or equal to outstanding data, TCP will not send data • In trace below, server is waiting for ACKs before sending more data Client test Server 271 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 135 2/10/2011 TCP Windowing • TCP uses the minimum size of the three windows – Congestion control window • Changes during connection • Internal to sending TCP • Not visible in real world • Lives in the “mind” of the sender – Receive buffer • May change during connection • User-configurable • Visible as WIN= in decodes – Send buffer test • Controlled by sending application • Not visible in decodes • Rarely the bottleneck 272 Congestion Control Window Receive Buffer In-Flight Data OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. TCP Congestion Control Window and Slow Start • Estimate of how much data the network can handle • Sendingg TCP increases the congestion g control window each time it receives an ACK – Congestion control window increases exponentially every round-trip time • Actually stored in bytes: 1460, 2920, 5840, 11680 • When TCP first starts transmitting data, it goes slowly – Sender starts by sending one segment, then waits for ACK test 273 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 136 2/10/2011 TCP Congestion Control Window in Slow Start Sender Receiver CWND = 1 Data (S1) ACK (S2) CWND = 2 Data (S2) Data (S3) ACK (S4) CWND = 3 Data (S4) Data (S5) Data (S6) CWND = 4 test CWND = 7 274 Data (S7) Data (S8) ACK (S8) Data (S9) Data (S10) Data (S10) (S11) ACK Data (S10) ACK (S12) OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Bulk Data Transfer • Congestion window is greater than receive window • TCP sends data as soon as it receives acknowledgements • In-flight g data is constant • In-flight data = outstanding data = unacknowledged bytes • Data transfer limited by receive window test When you see a pattern like this, blame TCP 275 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 137 2/10/2011 Review: TCP Windowing • TCP limits the amount of outstanding data • Minimum of three windows is used – Send window: How much data the sender can handle – Receive window: How much data the receiver can handle – Congestion control window: How much data the network can handle • Congestion window – Initially small (1, 2, or 4 segments) – Grows exponentially during slow start test 276 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. LAB 4 LAB 4 Examining TCP Behavior test 277 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 138 2/10/2011 Lab 4: Examining TCP Behavior • Use AppTransaction Xpert to examine TCP behavior • You will see – Connection opens – Slow start – Congestion control window • Raise your hand, and a TA will assist you with any questions • Starts on page 95 in your lab manual test 278 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Lab 4: Conclusion • TCP uses SEQ and ACK numbers to ensure reliable delivery • Minimum of three windows (congestion, receive, and send) determines whether TCP sends more data • Comparison of bulk and chatty applications – Window size and bandwidth determine throughput for bulk applications – Latency determines throughput for chatty applications test 279 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 139 2/10/2011 When to Suspect TCP as a Bottleneck • You should suspect TCP of causing problems when: – You cannot explain why an application is running slowly – The servers and network are adequately provisioned • Servers have low load • Sufficient bandwidth • Links not congested – The ACE Live™ Response Time Composition Chart shows • Large Payload Transfer (Xfer) Times • Large Retransmission Delays – AppTransaction Xpert ’s AppDoctor Summary of Delays tab shows • Large Network Transfer Effects test • Large Protocol delays – AppTransaction Xpert ’s AppDoctor Diagnoses point to • One or more TCP-related bottlenecks or potential bottlenecks 280 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Troubleshooting TCP with ACE Live - Excellent for focusing troubleshooting, monitoring TCP health • Where to start looking, where and when to capture • Measures connection resets, packet loss, long response times, and more ACE Live Response Time Composition Chart – Large Retransmission Delay • May be sign of packet loss • May also be unfiltered duplicate packets – Large Payload Transfer Times • May be TCP-related • May also indicate bandwidth, congestion or even server processing causes test 281 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 140 2/10/2011 Troubleshooting TCP with AppTransaction Xpert • AppDoctor provides quick way to check for TCP problems • Suspect TCP if the Summary of Delays shows – Large Network Transfer Effects – Large Protocol delays • Use diagnosis tab to identify which specific TCP-related issue affects your trace – Bottlenecks – Potential bottlenecks test 282 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Understanding Network Transfer Effects • Combination of three effects related to transferring data – Bandwidth, Congestion, Protocol • Bottleneck bandwidth is needed to separate these phenomena – Refine Network Effects key to identification of delays due to the TCP protocol test 283 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 141 2/10/2011 How to Identify TCP Problems • Look at the AppDoctor Diagnosis tab – Effect of Protocol – Effect of Network Transfer – Last 7 stats deal with TCP – Helps target the specific TCP issue • Look at decodes for long application transactions • Look in the network test and chart at the start end of long delays 284 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. How to Identify TCP Problems (cont.) • Double-click in the Summary of Delays for more information • Use separate drilldown screens for Protocol and Congestion test 285 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 142 2/10/2011 How to Identify TCP Problems (cont.) • Congested networks also can cause protocol delays • TCP implements congestion avoidance • Single-sided captures of receiving data may make congestion look like protocol delay or tier processing test 286 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Review: Analyzing TCP Behavior • Look for TCP problems in traces with large amount of protocol delay • Congestion delays also induce protocol delays • AppDoctor pp (forensic) ( ) – Separately diagnoses protocol and congestion – Diagnoses several TCP problems • QuickPredict (what if?) – Separately models protocol and congestion – Can simulate changing TCP Window Size test 287 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 143 2/10/2011 Review: ACE QuickPredict • Quick simulation engine – Combination of analytical and discrete event simulation – Uses analytical models of TCP behavior • Initially shows a pristine view of network • Allows you to study effects of the network on the application • Outputs graphs of response time vs. one of the following: – Bandwidth – Latency – Packet loss – Link utilization – TCP receive window size – NEW: WAN Acceleration parameters • QuickPredicttest Bar Charts supports asymmetric window sizes 288 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Review: QuickPredict Input and Output • Y axis is always response time • X axis is a range of values for one of: – Bandwidth, Latency, Packet Loss, Link Utilization, TCP Window size, and WAN Acceleration (NEW!) Change values of other variables to see impact on response time test 289 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 144 2/10/2011 Multi-User QuickPredict test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Multi-User QuickPredict (MUQP) • Single-user QuickPredict considers one client only • MUQP allows you to have more than one – Client and client site – Applications (either ACE traces or ACE Whiteboard traces) – Data centers • Output – Response time (including – delay components) – Utilization • Results – Average test – N-th percentile 291 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 145 2/10/2011 Inputs to MUQP • Deployed applications (defined in ACE and/or ACE Whiteboard files) • Network topology (number of client sites, clouds, and data centers) • Tier-pair p pparameters ((bandwidth, latency, y link utilization, window size, and packet loss) • Application deployment options (number of transactions per hour for each client site) test 292 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Outputs of MUQP • "Average-case" response times for applications • "Worst-case" results for access links (utilization) and applications (total response time and components of tier processing, latency, protocol, and congestion delay) test 293 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 146 2/10/2011 Instructor Demo of MUQP test 294 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Bandwidth-Delay Product test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 147 2/10/2011 Bandwidth-Delay Product • Bandwidth-delay product — Defined as amount of data (in bytes) that can be sent before ACK is received for the first unacknowledged bit of data – 2 x one-way latency x bandwidth of slowest link • To “fill the pipe,” TCP window size must be equal to bandwidth-delay product • If window is too small, TCP must wait for acknowledgment g before sendingg new data • Example below – Bandwidth: – Latency: – BWDP: – Receive Window: – Throughput limited to: 1,544 kbps = 193 KB/sec 250 ms = 0.250 sec 2 x .25 x 193 = 96.5 KB 8 KB (1/12 ideal size) 130 kbps (1/12 link speed) test 296 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Bandwidth-Delay Product Capacity = .096 s × 125 KB/s = 12 KB (8 segments) Bandwidth Delay Product = 2 × Capacity = 24 KB (16 segments) Bandwidth-Delay 96 ms Sender Receiver test 1 Mbps 297 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 148 2/10/2011 LAB 5 LAB 5 Troubleshooting a Slow File Transfer test 298 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Lab 5: Troubleshooting a Slow File Transfer • Overview – Computer manufacturer moving bulk data from Chicago to Taipei has dedicated T1 without any packet drops – Not seeing expected response time – You are a network engineer – Your manager: “It must be the network” – Contemplating (expensive) link upgrade • Objective – Determine the cause of slow response time – Propose and test a solution test • Starts on page 107 in your lab manual 299 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 149 2/10/2011 Lab 5: Conclusion • Simple TCP configuration change improved performance • Over same time period: p – Large receive window – Small receive window test • What is the downside of making this change? 300 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Changing the Receive Window • How do I change the receive window on my computer? – Windows • http://www.microsoft.com/technet/network/tcpip/default.mspx • http://www.microsoft.com/technet/itsolutions/network/deploy/depovg/tcpip 2k.mspx • Google search: windows tcp/ip implementation details – Solaris • http://www.sean.de/Solaris/soltune.html#water – Linux • http://www.speedguide.net/read_articles.php?id=121 • Default for modern operating systems is 64 KB test – Maximum advertised receive window value in TCP packet is 64 KB – How do we handle today’s large capacity, high latency pipes? • TCP Window Scaling 301 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 150 2/10/2011 TCP Window Scaling • TCP allows for WIN= field to be scaled by a power of 2 – Decodes will show WIN<<2=262140 – WIN<<x means multiply field by 2x • This setting is exchanged in TCP options at beginning of connection – Both sides must agree and understand this option test • What if a packet trace does not contain the opening of a connection? – AppTransaction Xpert cannot detect the presence of TCP Window scaling 302 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Detecting Packet Loss test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 151 2/10/2011 TCP and Packet Drops • Where do packets get dropped, and why? – Media collisions – Wireless signal interference – Congested networks • Endpoints must react to overloaded routers – Drops are a form of indirect communication between routers and senders • Routers have multiple strategies for keeping queue size in check – Tail drop – Random Early Detection (RED) test 304 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Packet Loss and Retransmissions Packet is dropped by the network and retransmitted Time between when it should have arrived and when it did arrive Controlled by TCP timers Counted C t d as P Protocol t lD Delay l Retransmission also invokes congestion control Slows down throughput Counted as Protocol Delay T = 0 1 2 3 4 5 6 7 8 9 10 Client A B B test Protocol Delay Server 0 1 2 3 4 5 6 7 8 9 10 305 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 152 2/10/2011 Drops and Retransmissions • TCP must make sure all bytes of a stream are received – TCP guarantees delivery by retransmitting packets presumed dropped • Two methods by which TCP will retransmit a dropped packet – Retransmission timeout • Sending TCP schedules a timer for outstanding packets • Timer is cancelled when the packet is acknowledged • If timer expires, packet is presumed missing, re-sent – Triple duplicate ACKs • ACK number indicates next expected packet • If receiving TCP does not get the expected packet, it ACKs every packet test • If receiving TCP does not get the expected packet, ACK number does not increase. • If sending TCP receives acknowledgements with the same ACK number, it knows a packet was likely lost 306 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. TCP One Loss: Retransmission Timeout Sender Receiver Data (S101) Data (S102) Data (S103) Data (S104) ACK ((S103)) Data (S105) ACK (S105) Data (S106) Data (S107) Data (S108) Retransmission timeout ACK (S106) ACK (S106) test Data (S106) ACK (S109) 307 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 153 2/10/2011 TCP One Loss: Triple Duplicate ACK Sender Receiver Data (S101) Data (S102) Data (S103) Data (S104) ACK (S103) Data (S105) Triple duplicate ACK and fast retransmit Data (S106) ACK (S104) Data (S107) ACK (S104) Data (S108) ACK ((S104)) test Because 104 was not received, Receiver does not change ACK number ACK (S104) Data (S104) ACK (S109) 308 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Cost of Loss Detection • Retransmission timeout – There are not enough ACKs for triple duplicate ACK – Cost: Timer (often large) plus time to send 1 packet – Congestion window decreases to one segment • Triple-duplicate ACKs – Cause fast retransmit – Cost: Time to send 1 packet Retransmission Triple-duplicate ACK test • Notice the send times 309 Retransmission OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 154 2/10/2011 Review: Detecting Packet Loss • Two mechanisms – Retransmission time-out (RTO) • Costly • Always works – Triple-duplicate ACK • Relatively efficient • Does not always happen • Chatty applications rarely “qualify” • Food for thought – Why acknowledge every packet during loss? – Why not quintuple-duplicate ACK? test – Why not single-duplicate ACK? 310 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Frozen Window test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 155 2/10/2011 TCP Frozen Window • Receiver stores packets in receive buffer • Advertised receive window “WIN=” reduced by data in buffer • Normal applications pp remove data quickly q y • What happens if: – A server is handling too many clients? – What if the server is running other applications? – What if a another thread is hogging the CPU? test 312 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. TCP Frozen Window (cont.) • As WIN decreases, buffer fills up, sender cannot send data • Connection becomes “frozen” • Sender will send pprobe packets p to see if window size has increased test 313 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 156 2/10/2011 TCP (Almost) Frozen Windows • As WIN decreases, sender cannot send data • What if the WIN is less than MSS? • Sender could send data, but often does not • Detection of “chilly windows” new to AppTransaction Xpert 16.0 test 314 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Drilling into Protocol (cont.) test • Frozen Windows: – Receiving WIN goes to zero – Sending side sends “probe” packets 315 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 157 2/10/2011 Nagle’s Algorithm test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Nagle’s Algorithm • Nagle’s Algorithm – TCP mechanism to reduce sending of many tiny messages – Sender may not send data unless a full, MSS-sized segment can be sent OR there is no unacknowledged data • Delayed Acknowledgement • Interaction of these two mechanisms sometimes causes problems • Ridiculous situation where both sides wait for the other to do something Client test A B B Server 317 200 ms OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 158 2/10/2011 Nagle’s Algorithm Example • Client sends segment A (less than MSS) • Client wants to send segment B, but needs to wait for ACK • Server is waitingg for another segment g before sendingg ACK Eventually, server sends ACK Client then sends segment B Process repeats… Client test A B B Server 318 200 ms OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Nagle’s Algorithm Example (cont.) • What causes the delay for this application? test 319 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 159 2/10/2011 Nagle’s Algorithm and Delayed ACK • To eliminate delays when sending consecutive small messages: – Rewrite the application • Sending multiple small consecutive messages is inefficient • Sometimes you can’t change the application – Disable Nagle’s algorithm • Done within the application • Effects only the application changed: (TCP_NODELAY socket option) • Not always possible to change an existing application – Acknowledge every packet • Practical real world alternative • On Windows: http://smallvoid.com/article/winnt-nagle-algorithm.html test • Warning: Applies to every application on the machine! – Interacts with congestion control window, affecting bulk applications 320 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. References • TCP in OPNET – Online documentation – Overview PDF • Model Library/Standard / TCP Model Usage Guide – ACE TCP Windowing Wi d i Bottleneck B ttl k online li documentation d t ti • Applications in OPNET – Several whitepapers including SMARTE • Help / Product Documentation / Modeling Methodologies / Methodologies and Case Studies • http://www.psc.edu/networking/projects/tcptune/ – Setting system wide TCP window sizes – Listing of TCP features for each operating system • http://www.ncsa.uiuc.edu/People/vwelch/net_perf/tcp_windows.html – Instructions on setting TCP window sizes within application code • http://www.sean.de/Solaris/soltune.html test – Tuning your TCP/IP stack on Solaris • TCP/IP Illustrated, Volume I, Richard Stevens 321 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 160 2/10/2011 Take-Away Points • Windowing causes TCP to wait before transmitting application data • Bulk data-transfer throughput often limited by TCP windowing • TCP frequently q y does not use the full available bandwidth • Packet loss can cause significant reduction in throughput • QuickPredict is a fast, easy, accurate way to predict application response time • Use DES to model and study detailed behavior • Simulate using the proper TCP machine settings Next step: Run homework labs for a greater understanding of TCP test 322 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. What’s New ?? test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 161 2/10/2011 What’s New in AppTransaction Xpert ?? • 64-Bit Support – Access more that 4GB of memory on 64-bit machines which allows you analyze much larger capture files • Windows 7 Professional (32-bit and 64-bit) • Capture Enhancements – Manage captures from ACE Live appliances within the Application Capture Manager. New ACE Live tab to quickly download and preview data – ACE Capture Agent 3.9 Ability to filter the traffic data by time range, IP addresses, and/or IP conversations. – Capture File Filters Greater flexibility in specifying hosts and ports. • Support for Riverbed Transparent Mode test – The AppTransaction Xpert import now supports capture files from Riverbed accelerators running in full or port transparency mode 324 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. What’s New in AppTransaction Xpert (cont.) • Streamlined Import Process – Quickly open capture files through the user interface without specifying any information (such as location, bandwidth, latency, etc.) • Bandwidth, Bandwidth Latency Latency, and Locations can be specified as needed – Automatic Filtering of Duplicate Packets and TSO (TCP Segmentation Offloading) – Simplified Manual Merge Workflow • QuickPredict Enhancements - WAN Acceleration Support – Support for WAN optimization technologies by allowing users to predict behavior for compression, caching, and application acceleration • HTTP Analysis Anal sis Enhancements test – Automatic detection of HTTP on non-standard ports (thus eliminating the need to perform redecodes or modifying config files) – Improved analysis for packet slicing and missing packets – HTTP-specific visualization of object response times in TreeView 325 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 162 2/10/2011 References test OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Additional Documentation • Online at: www.OPNET.com / Support / Support Center –Documentation –Release Notes • Also look at: http://userforums.opnet.com –Info on ALL OPNET products –Discussion Groups –Hints Hi & tricks i k test 327 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 163 2/10/2011 OPNET Training Courses • OPNET provides Instructor-led training for all of our products. –OPNET OPNET Training Facilities; • Bethesda, Cary, Dallas, Nashua, Santa Clara –On-site at YOUR facilities –On the Web via WebEx • For AppTransaction Xpert : –2-day Instructor-led-training test • Register for courses online at: www.opnet.com/Training 328 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Certification Program Become recognized as an expert user of OPNET products… Certification covers relevant topics such as: • • • • • Management / Administration Configuration Product Selection Product Operation Networking Knowledge Upon p successful completion: p • Letter test • Certificate • Individual certification ID • Valid for 2 years 329 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 164 2/10/2011 AppTransaction Xpert Online Training test 330 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Web Pages - Training Resource test 331 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 165 2/10/2011 OPNETWORK is the largest event of its kind, attracting thought leaders from industry, government, and academic communities from all over the world world, to advance best practices for: test • Application Performance Management • Network Engineering, Operations, and Planning • Network R&D RSVP at: www.opnet.com/opnetwork2010 332 332 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. Course Evaluation We’ll use this information to improve our courses and teaching methods. Please enter as much comment material as you’d like… the more info you add, the better we’ll get! Point your browser to: http://eval.opnet.com Please enter Instructor and Course info… – Course: Introduction to AppTransaction Xpert – Course Date: – Instructor Name: FREE Stuff !! test Thank You in advance for your participation! 333 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 166 2/10/2011 test 334 OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc. 167