2pm_1726_NetBackup 7.6 Best Practices Improving - VOX

Transcription

2pm_1726_NetBackup 7.6 Best Practices Improving - VOX
Session 1726 - NetBackup 7.6 Best
Practices: Improving Recovery Times
George Winter, Technical Product Manager
Reneé Carlisle, Sr. Product Manager
Session 1726 - Improving Recovery Times
1
Sample Agenda
1
Recovery Challenges
2
NetBackup Tools That Help Overcome Challenges
3
Deep Dive on VMware Recoveries
4
Final Thoughts
5
Questions
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
2
Recovery Challenges
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
3
Recovery Challenges
Optimize Type of Recovery
Slow Recovery Time (RTO)
Tighter Recovery Points (RPO)
• Single File Restores
• Full System Restores
• Full site restores
The larger the recovery scope, the
greater the cost
Tight Backup Window
• Backup can’t complete in the allotted time
• Data has outgrown the backup window
• Recovery can’t happen without a backup
• Recovery needs to be quicker
• Backups need to happen more
frequently
Data Loss
Downtime
Disaster Recovery requires data to be
available
• Trucking Tapes Has Risk and High
Operational Cost
• Array Based Replication Increases
Storage Cost and is the most
Expensive
• Data has to be available where you
need it
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
4
Your Recovery Time is dependent on:
Data Growth
• Data
How Archiving
much data do you have?
• Deduplication
How fast is it growing?
• Accelerator
Is it all equally important?
• Replication
Do you needDirector
instant access to
it?
• Use Parallel streams
CPU
CPU
• Offhost Backup
• How much compute
• Accelerator
power do you have?
• Replication
Director
• Is the load
too high
on the client?
• Use Appliances
Session 1726 - Improving Recovery Times
I/O
• Client Direct
•• How
fast can you read from
Accelerator
your source?
• Snapshots/RD
• How Fast can you write to
target?
• your
OptDupe/AIR
• Use Appliances
Network Bandwidth
• OptDupe
How much bandwidth do you
have?
• AIR
• Do you need to send it all offsite?
• Accelerator
• Where do you need to recover?
SYMANTEC VISION 2014
5
Leverage NetBackup to reduce Recovery Challenges
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
6
Leveraging the Appliance to meet your RTO
Reduce RTO, Improve DR, Decrease Backup Window
Meet RTO & RPO objectives with Appliances
5230 – 2.5.2
5220 – 2.5
Backup - Peak Throughput Client
Deduplication
100 Streams
30.85 TB/hr
23.66
TB/hr
Backup – Peak Throughput Target
Deduplication
100 Streams
8.33 TB/hr
3.85 TB/hr
Restore – 8 streams, 80 GB of data
331 MB/s
213 MB/s
Optimize Type of Recovery
Slow Recovery Time (RTO)
Tighter Recovery Points (RPO)
Data Loss
Tight Backup Window
•
30% Faster Peak
Client Dedupe
Performance
•
155% Faster Peak
Target Deduplication
•
50% Faster Restore
Speed
Disaster Recovery requires
data to be available
Downtime
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
7
Upgrade to take advantage of performance
improvements
Meet RTO & RPO objectives with Appliances
5230 –
2.5.x
Description
5230 – 2.6
Backup - Peak Throughput Client Deduplication
@98% 144 Streams
30.8 TB/hr
39.9 TB/hr
Backup – Peak Throughput Target All-In-One
Deduplication@98% 366 Streams
8 TB/hr
11.7 TB/hr
Restore – 4 streams
331 MB/s
538.98 MB/s
Optimize Type of Recovery
Slow Recovery Time (RTO)
Tighter Recovery Points (RPO)
Data Loss
Tight Backup Window
•
29% Faster Peak
Client Dedupe
Performance
•
46% Faster Peak
Target Deduplication
•
39% Faster Restore
Speed (4 streams)
•
38% Faster Restore
of 100th Backup
Disaster Recovery requires
data to be available
Downtime
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
8
Regarding my monthly weekend full backup of
a Linux client that mounts 13 TB SAN storage
that backs up to MSDP and then duplicates to
tape. Before the upgrade the duplication to
tape process would not complete till mid
week. After the upgrade it completed over
the weekend! Very satisfied with v7.6
duplication improvement.
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
9
Advanced Disk
Reduce RTO, Decrease Backup Window
Meet RTO objectives with Advanced Disk
Adv Disk Pool Gold
• Allows for fast staging
area for increased RTO
Adv Disk Pool Silver
NFS
Mounts
• Use commodity disk that
can span physical
boundaries
• Allows classification of
data for increased
protection of more critical
data
Disk Volumes
NetBackup Media Servers function as both storage servers and data mover
Optimize Type of Recovery
Slow Recovery Time (RTO)
Tighter Recovery Points (RPO)
Data Loss
Tight Backup Window
Downtime
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
10
Storage Lifecycle Policies
Reduce RPO/RTO, Decrease Backup Window, Improve DR
Meet RPO & RTO objectives with Storage Lifecycle Policies
backup job 1
backup job 2
backup to
tape, retain 6
months
bronze
lifecycle
policy:
silver
lifecycle
policy:
backup to
appliance,
retain 2
months
gold
lifecycle
policy:
backup to disk,
retain 3 weeks
backup job 3
• Automatically move
data through its
lifecycle
duplicate tape,
vault offsite &
retain 6
months
Write to
appliance
retain onsite 2
months
write to tape,
vault offsite,
retain 6
months
backup job 4
Optimize Type of Recovery
Slow Recovery Time (RTO)
Tighter Recovery Points (RPO)
Data Loss
Tight Backup Window
• Ensure data is stored
in the right place for
the right time
• Ensure that you
always have a copy of
data available that
meets your retention
requirements
Disaster Recovery requires
data to be available
Downtime
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
11
Auto Image Replication (A.I.R.)
Reduce RPO/RTO, Decrease Backup Window, Improve DR
Meet RTO & RPO objectives with Auto Image Replication
Production Data Center #1
Domain B
Branch office
Domain C
Remote office
Domain D
Production Data Center #2
Domain A
Optimize Type of Recovery
Slow Recovery Time (RTO)
Tighter Recovery Points (RPO)
Data Loss
Tight Backup Window
• Data is off site as soon
as your backup policy
completes
• Data and applications
are backed up, deduped,
and replicated
immediately
• Data and applications
are available for restore
- NOW
• No tapes to search for
and load
Disaster Recovery requires
data to be available
Downtime
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
12
AIR jobs from MSDP would normally take all
weekend, with jobs still running on Monday
morning. After the 7.6 upgrade, the first
weekend AIR jobs were all complete by
Sunday noon. This is due to the MSDP
rehydration performance improvements in
7.6
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
13
SAN Client
Reduce RTO, Decrease Backup Window
Meet RTO objectives with NetBackup SAN Client
• Fast SAN Backup–
150 MB/sec backups,
up to 500 MB/sec
aggregate through a
media server
SAN Clients
load-balanced
media servers
fc
fc
fc
fc
fc
fc
fc
fc
fc
fc
os
fc
os
fc
fc
OpenStorage
Devices
Adv Disk Pool
Optimize Type of Recovery
Slow Recovery Time (RTO)
Tighter Recovery Points (RPO)
Data Loss
• Remove Backup
Impact from the LAN –
dedicated SAN backup
fc
NetBackup
Appliance
• High Availability – can
configure redundant
Fibre Channel paths
Tight Backup Window
Downtime
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
14
Accelerator: Files and Folders and VMware
Optimize Recovery, Reduce RPO/RTO, Decrease Backup Window
Meet RTO & RPO objectives with Accelerator
• Decreased backup
time allows for more
frequent full backups
Master Server
Client
NBU
Catalog
Application
1
File System
NTFS Change Journal
Optimize Type of Recovery
Slow Recovery Time (RTO)
Tighter Recovery Points (RPO)
Data Loss
2
Media Server
Synthesis
Engine
Dedupe
Engine
Track Log
5
3
4
Backup
engine
• low I/O and CPU cost
on client, network
bandwidth and
storage cost
decreases CapEx
• Reduce RTO by
recovering from a full
rather than a series of
incremental backups
Tight Backup Window
Downtime
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
15
Accelerator had actually meant it’s signature
quote of speed up backup with 100X, indeed it
is here in our case (182%). Seeing at
improvements NetBackup had made, would
strongly recommend to all existing customers
to upgrade their environment on this release.
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
16
FlashBackup
Optimize Recovery, Reduce RPO/RTO, Decrease Backup Window
Meet RTO objectives with FlashBackup
• Combines the speed of
raw-partition backups with
the ability to restore
individual files
• Supports multiple data
streams
• Best for file systems that
contain a large number of
files where most of the file
system blocks are
allocated and have a high
change rate
Optimize Type of Recovery
Slow Recovery Time (RTO)
Tighter Recovery Points (RPO)
Data Loss
Tight Backup Window
Downtime
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
17
Replication Director
Optimize Recovery, Reduce RPO/RTO, Decrease Backup Window,
Improve DR
Meet RPO & RTO objectives with Replication Director
NetBackup
Admin
Console
NAS File Services
Oracle
on NFS
DB2
SAP
Catalog
VMware
on NFS
MS Exchange, SQL
Server (on
VMware)
Hyper V
• Application
consistency for
supported workloads
Snapshot copies and replication fully integrated into
backup data life-cycle
Streaming Tape Backup
NDMP Tape Backup
SnapMirror/SnapVault
Tape
NDMP Disk Backup
Snapshot copies on
primary offer low impact
Optimize Type of Recovery
Slow Recovery Time (RTO)
Tighter Recovery Points (RPO)
Data Loss
Storage efficient blocklevel incremental
Snapshot replication
Tight Backup Window
• Protect entire volumes
of data with Hardware
Snapshots, while still
maintaining GRT
Disk
Leverage Accelerator with
final disk backup using
Windows Policy
• Manage replication
and long-term copies
from a single policy so
data is where you
want it, when you
need it
Disaster Recovery requires
data to be available
Downtime
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
18
Application Protection with NetBackup Agents
Optimize Recovery, Reduce RTO, Decrease Backup Window
Meet RTO objectives with Application Agents
Application Integration API
• Recover entire
database or inidividual
components from a
single backup
Application Protection Policies
Local Snapshot
Instant Recovery
Off Host
Backup Transport
Network
SAN
Storage Target
Optimize Type of Recovery
Slow Recovery Time (RTO)
Tighter Recovery Points (RPO)
Data Loss
• Eliminate 2-step
recovery process
required by database
dumps
• Leverage transport
and storage options
that match your RPO
and RTO SLAs
Tight Backup Window
Downtime
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
19
Instant Recovery with NetBackup Snapshot Client
Reduce RPO/RTO, Decrease Backup Window
Meet RPO objectives with Instant Recovery using Snapshot Client
• Meet recovery SLAs
by enabling
application instant
recovery
Enterprise Client &
Data base Agents
Primary
Snapshots
Tape
Media Servers
• Leverage array-based
snapshots or Veritas
Storage Foundations
• Increase reliability by
removing manual
process
Disk
Optimize Type of Recovery
Slow Recovery Time (RTO)
Tighter Recovery Points (RPO)
Data Loss
Tight Backup Window
Downtime
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
20
Bare Metal Restore
Optimize Recovery, Reduce RTO
Meet RTO & RPO objectives with Accelerator
•
Consistent and
functional system
recovery
•
Integrated, easy to
manage and
administer
•
Scalable
•
High degree of
recovery automation
•
Supports Dedupe
and Accelerator
•
Provides recovery
flexibility
Step
9.
Reboot
8.
Load
tape(s) and
restore
Reboot
7.
Server Recovery Time
6.
Reload
backup
software
5.
Reboot
4.
Step
Reload OS
3.
3.
Reboot
2.
1.
Collect all
media
Click “Prepare
to Restore”
Repair
hardware
1.
Repair
hardware
Traditional
Recovery
Optimize Type of Recovery
Reboot
2.
Bare Metal Restore
Slow Recovery Time (RTO)
Tighter Recovery Points (RPO)
Data Loss
Disaster Recovery requires
data to be available
Downtime
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
21
Automated Physical to Virtual Conversion
Reduce RPO/RTO
Meet RTO & RPO objectives with VMware P2V
Master server
Media server
• Eliminate need for
stand by hardware –
decrease CapEx
Storage containing backup
Data store that contains
• Have instant access to
servers without
recovery
converted virtual instances
Virtual Instance Convertor
VMware ESX or
(NB-Proxy Host)
vCenter server
Optimize Type of Recovery
Slow Recovery Time (RTO)
Tighter Recovery Points (RPO)
Data Loss
Virtual Environment Setup
Disaster Recovery requires
data to be available
Downtime
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
22
VMware Instant Recovery
Optimize Recovery, Reduce RPO/RTO
Meet RTO & RPO objectives with Accelerator
•
Instantly power on
any protected VM
from disk backup
target
• No need to
restore VM first
•
Uses standard
NetBackup backup
images
•
Once powered on,
VM is 100% available
ESX/ESXi
NAS
NetBackup NFS Datastore
SAN
Optimize Type of Recovery
Slow Recovery Time (RTO)
Tighter Recovery Points (RPO)
Data Loss
Disaster Recovery requires
data to be available
Downtime
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
23
Leverage the power of combining solutions
Optimize Recovery, Reduce RPO/RTO, Decrease Backup Window
Meet RTO & RPO & DR objectives with combined technologies
NBU Master
Domain A
NBU Clients
Physical/Virtual
• RD + Accelerator
Import image
Client System
Configuration
backup
• AIR + P2V
NB client
which drives
conversion
NBU BMR
Master
Domain B
NBU
Media
Server
Create Clients
Virtual Instances
Import client
system info
Backup
Device Notifies
NBU
Image
NBU Media
Server
OST Appliance
or PureDisk
Image
OST Appliance
or PureDisk
• VADP + Accelerator +
Appliances + AIR +
VIR
Virtual Environment
Hypervisor Server:
(VMWARE ESX or
HyperV)
OST Optimized
Duplication
Production Data Center
Optimize Type of Recovery
DR Domain
Slow Recovery Time (RTO)
Tighter Recovery Points (RPO)
Data Loss
Tight Backup Window
Disaster Recovery requires
data to be available
Downtime
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
24
The nice thing about NetBackup 7.5 is that with
deduplication, optimized image replication,
and NetBackup Accelerator, we have some file
servers that have 600 gigabytes of SharePoint
data on them and a full backup is done in 40
minutes that would take two days before.
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
25
Deep Dive on VMware Restores
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
26
VMware Restore Considerations
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
27
General Restore Performance Thoughts
• I/O – Spinning Disk
– Reading data from disk is easy
– Writing data to disk is hard
– Full disks (> 80%) slow this process
• I/O – Tape (not VTL)
– Reading data from tape is hard (multiplexed)
– Writing data to tape is easy
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
28
VMware Restores With vStorage API for Data Protection
• Backups based on VADP
– VMDK itself is *not* backed up – common misconception
– Data *inside* VMDK is backed up
– This provides ability to reformat VMDK provisioning at restore
– Backup more efficient – skips unused space (NBU adds efficiency too)
• NetBackup with VADP enables additional restore capabilities
– Single file (e.g. Word doc) restores from image (VMDK) backup
– Database object level restore from image (VMDK) backup
– Physical to Virtual
– Virtual to Virtual (somewhat manual)
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
29
Why Are VMware Restores Slower Than Backups?
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
30
VMware VMDK Restore Transport Modes
• NBD Transport – Restores
–
–
–
–
–
VMkernel port bandwidth limit
First stream always fastest
First stream ≈ 100 MB/sec (10 GbE)
Subsequent (simultaneous) streams slower
No way around this with NBD transfers
• SAN Transport – Restores
– No VMkernel port QoS limitation
– Can be fastest traditional restore performance
• HotAdd - Restores
– VMkernel port not used
– Can be similar to SAN restore speeds
– Dependent on available ESXi host resources
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
31
What Happens During VM Image (VMDK) Restore?
1) VM image restore initiated
2) New VM registered in vCenter
(Ever try to boot the VM at this point?)
3) New VMDK is created
4) All space inside VMDK must be zeroed
(Note that at this point zero data has been restored)
5) NetBackup begins restoring VM data
This explains why a tape may be quickly mounted with delay
before data written to tape (or any storage unit)
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
32
How VMDK Provisioning Impacts Restore Times
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
33
The Unvarnished Truth: Thin Provisioned VMDK Restore
1. An empty 16MB VMDK is first created
2. This 16MB chunk is “zeroed”
– This step mandatory
Restored
Data
Restored
Data
3. Data restored to this chunk
4. Process repeated until all data restored
– Can’t determine final size of VMDK
Restored
Data
• Choice of thin or thick provisioned VMDK
impacts restore time
Restored
Data
• Restores take twice as long as backups (approx)
Restored
Data
• SAN restores impacted by busy vCenter
– All restore instructions channeled through vCenter
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
16 MB
Thin VMDK
34
EagerZeroedThick Provisioned VMDK Restore
• VMDK is created using 100% required space
• Entire VMDK is “zeroed”
• Data restore process begins
• Thin or Thick Provisioned? Which is faster?
Restored
Data
• Small percentage of restore VMDK data
– Thin provision faster
• Large percentage of restore VMDK data
– Thick provisioned probably faster
• Common to have choice dictated by VM admin
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
Thick VMDK
35
An Alternative To This Restore Process
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
36
Instant Recovery For VMware (IRV)
• Best possible RTO - VM or group of VMs instantly available
• Reverse traditional restore process:
– Traditional restore: 1) Restore VM (hours) 2) Boot VM (minutes)
– IRV restore: 1) Boot VM (minutes) 2) Restore VM (hours)
• Engineering tests indicate 30 second boot times (YMMV)
• Works with any disk based NetBackup target
– Includes MSDP and NetBackup Appliance
• No change to NetBackup disk backup image is required
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
37
VMware Instant Recovery in NBU 7.6
• Instantly power on any protected
VM from disk backup target
– No need to restore VM first
• Uses standard NetBackup backup
images
ESX/ESXi
– No need to change any backup
process
NAS
• Support with all Symantec disk
based solutions
NetBackup NFS Datastore
– Basic disk, Advanced disk, PDDO,
MSDP, NetBackup appliance
• Once powered on, VM is 100%
available
SAN
– After power-on, VM disks transferred
to ESXi storage (Storage VMotion)
– Storage VMotion ensures no
disruption of service
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
38
NetBackup Instant Recovery for VMware
Process Overview
Session 1726 - Improving Recovery Times
39
Instant Recovery for VMware Configuration Notes
VMware ESXi
LAN
NetBackup Appliance
• Zero modifications require to either
NetBackup or VMware environments
ESXi Datastores
• Works with NetBackup 7.5 images and vSphere 5.0 (or later)
• All ESXi Datastore types are supported (NFS, SAN, iSCSI, DAS)
• Works with NetBackup Appliance, MSDP and Adv Disk
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
40
Instant Recovery for VMware Configuration Notes
VMware ESXi
LAN
NetBackup Appliance
ESXi Datastores
Let’s See How Instant Recovery for VMware Works…
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
41
Instant Recovery for VMware Process
VMware ESXi
LAN
NetBackup Appliance
1
NetBackup disk storage is
provisioned as read-only NFS
Datastore
ESXi Datastores
1
VM1
Note that VM1 is currently
located on NetBackup disk
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
Temp NetBackup
Read Only
Datastore
42
Instant Recovery for VMware Process
2
2
VM is automatically
created and registered
in vCenter
VMware ESXi
LAN
NetBackup Appliance
ESXi Datastores
VM1
Temp NetBackup
Read Only
Datastore
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
43
Instant Recovery for VMware Process
3
VM1 is now powered on
REDO location automatically configured
All changes to VM1 are captured in REDO
VMware ESXi
LAN
3
NetBackup Appliance
REDO
ESXi Datastores
VM1
Temp NetBackup
Read Only
Datastore
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
44
Instant Recovery for VMware Process
VMware ESXi
LAN
NetBackup Appliance
REDO
At this point VM1 is 100% accessible
to all users
ESXi Datastores
All changes that occur are safely
captured in REDO
This entire process can take less
than 60 seconds
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
VM1
Temp NetBackup
Read Only
Datastore
45
Instant Recovery for VMware Configuration Notes
VMware ESXi
LAN
NetBackup Appliance
4
4
REDO
Storage VMotion is now initiated
VMDK(s) are copied to final
destination
During this process, VM1 is still
100% accessible
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
VM1
Temp NetBackup
Read Only
Datastore
46
Instant Recovery for VMware Configuration Notes
VMware ESXi
LAN
NetBackup Appliance
5
5
VM1
REDO
REDO is automatically consolidated
into VM1
All changes that occurred during this
process are automatically retained
Temp NetBackup
Read Only
Datastore
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
47
Instant Recovery for VMware Configuration Notes
VMware ESXi
LAN
NetBackup Appliance
VM1
ESXi Datastores
6
Instant Recovery process is now
complete
Temporary Datastore is removed
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
6
Temp NetBackup
Read Only
Datastore
48
IRV Restore Method Comparison – 6 TB VM
• Standard restore method = 25h:01m
• IRV restore method = 3m:22s
446x faster restore with NetBackup IRV
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
49
Final Thoughts On Improving Recovery
• Know what kind of recovery you need to do
• Understand the bottle necks in your environment
• Use Appliances
• Keep up with NetBackup upgrades
• Use the right NetBackup feature for your recovery SLA
• Restores are typically slower than backups - plan accordingly
• Virtualization provides additional restore options over physical
backups
• Be aware of VMware created limits for certain restore types
• VMDK provisioning and restore transport selection will impact
restore performance
Session 1726 - Improving Recovery Times
SYMANTEC VISION 2014
50
Thank you!
YOUR FEEDBACK IS VALUABLE TO US!
Please take a few minutes to fill out the short session survey available on
the mobile app—the survey will be available shortly after the session
ends. Watch for and complete the more extensive post-event survey that
will arrive via email a few days after the conference.
To download the app, go to https://vision2014.quickmobile.com or search
for Vision 2014 in the iTunes or Android stores.
Session 1726 - Improving Recovery Times
51