Lab presentation - Observing and Optimizing your Application with

Transcription

Observing and Optimizing your
Application with DTrace
Angelo Rajadurai
Principal Software Engineer
Oracle
Learn how to use DTrace for observing
applications. We will cover a wide range
of applications in this lab.
You will have opportunity to look at AMP
stack, scripting languages and some
Web 2.0 technologies
2
How we will work?
 We will spend the first hour and a half in
presentations
 The presentation covers
✦
✦
✦
✦
Basics of DTrace
Demo of D-scripts
DTrace Design Patters - developing D-scripts
DTrace for performance tuning
 You will have two hours work on the labs.
 Labs will be available on AWS
 Let me know if you need access
 Feel free to ask for help
3
DTrace lab site
http://dtracehol.com/
4
Slides are available as
http://dtracehol.com/OSCON-DTrace.pdf
5
Google Group for Lab
http://groups.google.com/group/dtracehol
6
Why Dynamic Tracing
7
Today’s application
python
Presentation
php
JSP/Javascript/Ajax/JavaFX
perl
Container Glassfish/J2EE
shell
Database
MySQL/Oracle/Postgres
C
C++
Frameworks
ruby
Apps run
in VM
Native
Apps
Operating System Solaris/Linux/OS X/Windows
8
Traditional Tools
• Development Debugging tools - Tightly bound to
programing language, Framework or Application
• System Tools (vmstat mpstat iostat) - Get system
wide view but no ability to drill down to workload
• truss pstack - not great for going beyond single
process views
• mdb(1) - Typically used as postmortem rather than
live analysis
9
’d
“I
r: e
e
p as
o
l
e tab
v
De da
e
h
t
k
as
n”
i
m
ad
“M
ay
D
be ata
th bas
e
e
w i m Ad
ll idd mi
kn le n:
o w wa
re
”
gu
ys
The Blame Game
“I
do Sy
n’t sa
De kno dmi
ve w! n
lop A
er sk
th
”
e
t: n”
r
e mi
p
Ex sad
re Sy
a
w he
e
l
t
d
id sk
M a
ts
e
“L
10
Static Instrumentation
Stop App
Restart
w/ flags
Add Code
Custom Instruments
Prebuilt Instruments
Instrument
Compile
Validate
Run in
production
Process
Collect info
11
Debugging Transient Problems
In production!
Please let me
observe a live
system.
12
Dynamic Tracing
Application Logic
Frameworks
Presentation
c
c+
ph
perl
shel
Container
Glassfish/
Database
rub
JSP/Javascript/Ajax
pytho
12::SELECT id, name, symbol, conversion_rate FROM currencies WHERE
status = 'Active' and deleted = 0
12::SELECT category, name, value FROM config
12::SELECT id FROM outbound_email WHERE type = 'system'
Native
Operating System Solaris/Linux/OS X/
13
So what is Dynamic Tracing?
• Turn ON and OFF dynamically
• Probe any arbitrary location
• not just pre-defined probe points
• Ability to collect arbitrary data
• Overhead of probing is small
• No Overhead for disabled probes
14
Answer!
DTrace
15
The D - Language
Basics
of
Dynamic Instrumentation
16
D - Scripts
probe
provider
/
: module :
probe
predicate
:
where
name
/
when
{
actions
do what
}
17
DTrace command line
# dtrace -n probe’/predicate/{action}’
18
demo
19
Aggregation
dtrace -P syscall
=> mmap
5 <= mmap
5 => munmap
5 <= munmap
5 => setcontext
5 <= setcontext
5 => getrlimit
5 <= getrlimit
5 => getpid
5 <= getpid
5 => setcontext
5 <= setcontext
5 => sysi86
5 <= sysi86
5 => brk
5 <= brk
5 => brk
5 <= brk
5 => xstat
5 <= xstat
5 => resolvepath
5 <= resolvepath
5 => open
5 <= open
5 => mmap
5 <= mmap
5 => mmap
5 <= mmap
5 => mmap
5 <= mmap
5 => mmap
5 <= mmap
5 => munmap
5 <= munmap
5 => memcntl
5 <= memcntl
5 => close
5 <= close
5 => xstat
5 <= xstat
5 => resolvepath
5 <= resolvepath
5 => open
5 <= open
5 => mmap
5 <= mmap
5 => mmap
5 <= mmap
5 => mmap
5 <= mmap
5 => mmap
5 <= mmap
5 => mmap
5 <= mmap
5 => munmap
5 <= munmap
5 => memcntl
5 <= memcntl
5 => close
5 <= close
5 => munmap
5 <= munmap
5 => ioctl
5 <= ioctl
5 => ioctl
5 <= ioctl
5 => brk
5 <= brk
5 => brk
5 <= brk
5 => fsat
5 <= fsat
5 => fcntl
5 <= fcntl
5 => fstat64
5 <= fstat64
5 => getdents64
5 <= getdents64
5 => getdents64
5 <= getdents64
5 => close
5 <= close
5 => ioctl
5 <= ioctl
5 => fstat64
5 <= fstat64
5 => write
5 <= write
5 => close
5 <= close
5 => close
5 <= close
5 => rexit
mmap
mmap
munmap
munmap
setcontext
setcontext
getrlimit
getrlimit
getpid
getpid
setcontext
setcontext
sysi86
sysi86
brk
brk
brk
brk
xstat
xstat
resolvepath
resolvepath
open
open
mmap
mmap
mmap
mmap
mmap
mmap
mmap
mmap
munmap
munmap
memcntl
memcntl
close
close
xstat
xstat
resolvepath
resolvepath
open
open
mmap
mmap
mmap
mmap
mmap
mmap
mmap
mmap
mmap
mmap
munmap
munmap
memcntl
memcntl
close
close
munmap
munmap
ioctl
ioctl
ioctl
ioctl
brk
brk
brk
brk
fsat
fsat
fcntl
fcntl
fstat64
fstat64
getdents64
getdents64
getdents64
getdents64
close
close
ioctl
ioctl
fstat64
fstat64
write
write
close
close
close
close
rexit
rexit
fcntl
fsat
getpid
getrlimit
sysi86
write
fstat64
getdents64
memcntl
open
resolvepath
setcontext
xstat
ioctl
brk
munmap
close
mmap
1
2
2
2
2
2
2
4
4
4
4
4
4
4
6
8
8
10
20
dtrace -P syscall’{@[probefunc]=count()}’
20
Demo
21
DTrace providers
Other
Providers
DTrace
core provider
ip
tcp
udp
java
perl
ruby
python
javascript
proc
fbt
DTrace
core
cpc
mib
PID
FC
sh
vminfo nfsv4
IO
syscall sysinfo
USDT
nfsv3
profile
plockstat
mysql
lockstat
postgres
kerberose
iSCSI
22
DTrace for your Day Job
• Pre-built D-scripts
• D-Script Design Patters
• Framework for developing D-scripts
23
D-Script Collections
• Excellent collection of D-script
• DTraceToolKit
• Docs/Examples - this directory has examples for each script.
• DTrace and IDE integration
• Project D-Light
• DTrace visualization
• Chime & DTrace Java API
24
DTrace Design Patterns
• Simplify D-script development
• DTrace design Patterns
•
•
•
•
•
•
The event trace pattern
The count pattern
What’s in between pattern
Time spent pattern
Profile Pattern
Call flow pattern
25
Event Trace Pattern
• Good for rare events
• Samples
dtrace -n swapin’{trace(execname)}’
syscall::read:entry
/strstr(fds[arg0].fi_pathname,$$1)!=0/
{
printf(“%s read from %s\n”,execname, $$1);
}
26
Count Pattern
• Good for collecting summary of events
• Samples
dtrace -n on-cpu’{@[execname]=count()}’
syscall:::entry
/execname==$$1/
{
@[probefunc]=count();
}
27
What’s In Between?
•
Good for drilling down into details
pid$1::$$2:entry
{
self->traceme=1;
}
pid$1:::method-entry
/self->traceme/
{
@[probefunc]=count();
}
pid$1::$$2:return
{
self->traceme=0;
}
28
Time Spent Pattern
• Find time spent in functions
syscall:::entry
{
self->ts=timestamp;
}
syscall:::return
/self->ts/
{
@[probefunc]=sum(timestamp-self->ts);
self->ts=0;
}
29
Profile Pattern
• An inexpensive way to find offending method
•
See hotuser & hotkernel scripts in DTraceToolkit
dtrace -n profile-101’/arg1/{@[ufunc(arg1),caller]=count()}’
30
DTrace Performance Tuning
• DTrace a great tool for perf tuning
• Ability to observe as problem happens
• Correlate various layers
• Ability to drill down to root cause
31
Gotcha’s
• Probe effect
• DTrace safe for production
• This does not mean no effect on performance
• DTrace is not pixie dust
• It can answer questions but
• Does not know what Questions to ask
• Does not interpret the answer
32
Approaching Performance
• Before you start using DTrace
•
•
•
•
•
•
Make sure you know and agree on the problem statement
• “System is Slow” is not a problem statement
Find out what is “known” - but verify
Is problem reproducible?
Can you split problem into CPU, IO & Network?
Are application instrumentation available?
Are other non-dtrace tools more appropriate?
33
Solaris Tool chest
Proc tools
cputrack - per-processor hw counters
pargs – process arguments
pflags – process flags
pcred – process credentials
pldd – process's library dependencies
plockstat – process lock statistics
psig – process signal disposition
pstack – process stack dump
pmap – process memory map
pfiles – open files and names
prstat – process statistics
ptree – process tree
ptime – process microstate times
pwdx – process working directory
pgrep – grep for processes
pkill – kill processes list
pstop – stop processes
prun – start processes
prctl – view/set process resources
pwait – wait for process
preap – reap a zombie process
Tracing & Debugging
abitrace - trace ABI interfaces
mdb – debug process/kernel
truss – trace function and system calls
System Statistics
acctcom – process accounting
busstat – Bus hardware counters
cpustat – CPU hardware counters
iostat – IO & NFS statistics
kstat – display kernel statistics
mpstat – processor statistics
netstat – network statistics
nfsstat – nfs server stats
sar – kitchen sink utility
vmstat – virtual memory stats
Toolkits
DTraceToolkit – DTrace Tools
K9Toolkit – perl perf tools
nicstat – network stats
34
System Wide Issues
• Start with vmstat, mpstat, iostat, etc.
•
Drill down with DTrace
• Each of the columns has DTrace probe equiv.
• Use “Count Design Pattern” to get summary
35
mpstat - dtrace drill down
• minf (Minor faults)
•
•
•
dtrace -n as_fault’{@[execname]=count()}’
majf (Major faults)
dtrace -n maj_fault’{@[execname]=count()}’
xcal (Cross calls)
dtrace -n xcalls’{@[execname]=count()}’
intr and ithr (Interrupts)
#!/usr/sbin/dtrace -qs
interrupt-start
{
this->name=stringof(`devnamesp[((struct dev_info *)arg0)->devi_major].dn_name);
@[this->name]=count();
}
or look at DTraceToolkit inttimes
36
mpstat - dtrace drill down (con’t…)
• csw (Context switches)
•
•
dtrace -n pswitch’{@[execname]=count()}’
icsw (Involuntary context switch)
dtrace -n inv_swtch’{@[execname]=count()}’
smtx (Spin on mutexes)
dtrace -n adative-spin,adaptive-block’{@[execname]=count()}’
• migr (Thread migration)
off-cpu
{ self->cpu=cpu; }
on-cpu
{ @[execname,self->cpu,cpu]=count(); self->cpu=0; }
37
mpstat - dtrace drill down (con’t…)
• usr/sys/wt/idle (CPU utilization)
on-cpu
{
self->ts=timestamp;
}
off-cpu
/self->ts/
{
@[execname]=sum(timestamp - self->ts);
self->ts=0;
}
38
vmstat - dtrace drill down
• re (Page reclaim)
•
•
•
•
•
dtrace -n page_reclaim’{@[execname]=count()}’
epi (Executable pages paged in)
dtrace -n execpgin’{@[execname]=count()}’
api (Anonymous pages paged in)
dtrace -n anonpgin’{@[execname]=count()}’
apo (Anonymous pages paged out)
dtrace -n anonpgout’{@[execname]=count()}’
fpi (Filesystem pages paged in)
dtrace -n fspgin’{@[execname]=count()}’
fpo (Filesystem pages paged out)
dtrace -n fspgout’{@[execname]=count()}’
39
iostat - dtrace drill down
• Look at the IO provider!
• io:::start & io:::done - Sync IO
• io:::wait-start & io:::wait-done - ASync IO
• DTraceToolkit also has some nice scripts
• iosnoop - snoop I/O events as they occur
• iotop - display top disk I/O events by process
• iofileb.d - I/O bytes by file and process
• iofile.d - I/O wait time by file and process
• iopattern - print disk I/O pattern
• iopending - graph number of pending disk events
40
Process Level Observation
• DTrace can give some excellent insight into processes
• pid provider
• Get into the heart of the process
• Just don’t go overboard (pid:::entry - is normally a bad idea)
• profile provider
• easy low cost way to get good information!
41
Hot Functions & Hot Libs
• Finding hot user function is easy
•
•
dtrace -n profile-1001’/arg/{@[ufunc(arg1)=count()}’
So is hot user libraries
dtrace -n profile-1001’/arg/{@[umod(arg1)=count()}’
You can even find hot callers
dtrace -n profile-1001’/arg/{@[ufunc(arg1),ufunc(ucaller)=count()}’
• What is the hottest non-idle loop, kernel func?
dtrace -n profile-1001'/arg0 && curthread->t_pri!=-1/{@[func(arg0)]=count()}'
• So is hot kernel module
dtrace -n profile-1001'/arg0 && curthread->t_pri!=-1/{@[mod(arg0)]=count()}'
42
Method Count / Time Spent
•
Counting methods is easy (may have big probe effect)
•
Time spent in methods
dtrace -n pid$1:libc::entry’{@[probefunc]=count()}’
pid$1:libc::entry
{
self->ts[probefunc]=timestamp;
}
pid$1:libc::return
/self->ts[probefunc]/
{
@[probefunc]=sum(timestamp - self->ts[probefunc]);
self->ts[probefunc]=0;
}
43
Much More Complicated
Call flow tracking
pid$1:::return
/self->ts && probefunc==$$2/
#pragma D option strsize=204800
{
pid$1::$$2:entry
self->cf=strjoin(strjoin(self->cf,"\n<-"), probefunc);
{
@t[self->cf]=sum(timestamp - self->ts);
self->ts=timestamp;
@i[self->cf]=min(timestamp - self->ts);
self->cf=strjoin("->", probefunc);
@a[self->cf]=max(timestamp - self->ts);
self->ats[probefunc]=timestamp;
@v[self->cf]=avg(timestamp - self->ts);
}
@c[self->cf]=count();
pid$1:::entry
self->cf=0;
/self->ts && probefunc!=$$2/
self->ts=0;
{
@ac[probefunc]=count();
self->cf=strjoin(self->cf,strjoin("\n->",probefunc));
@at[probefunc]=sum(timestamp - self->ats[probefunc]);
self->ats[probefunc]=timestamp;
self->ats[probefunc]=0;
}
}
pid$1:::return
END
/self->ts && probefunc!=$$2/
{
self->cf=strjoin(self->cf,strjoin("\n<-",probefunc));
@ac[probefunc]=count();
@at[probefunc]=sum(timestamp - self->ats[probefunc]);
self->ats[probefunc]=0;
{
printa("%s\n Total Time: %@d :: Fastest: %@d ::
Slowest: %@d :: Average: %@d :: Count: %@d\n\n\n",@t, @i,
@a, @v, @c);
%s\n",$$1);
printf("\n\nSUMMARY of all methods called from
printf("%-7s %-15s %s\n","COUNT",
"TOTAL","CLASS.METHOD" );
printa("%@-7d %@-15d %s\n",@ac, @at);
}
}
44
DTrace Hands-on-lab
45
Quick Walk-through

The lab will be run on a instance on Amazon EC2.

Each one of you will have your own instance

If you have not logged in, please log in with the details
on your one pager.

Goto http://dtracehol.com/

This will have the instructions for the lab

All necessary software and lab files are already
installed on each instance.
46
Secure Global Desktop

Secure Global Desktop allows you to experience
OpenSolaris

DOIP - Display Over IP

Simple Web based interface
47
Bandwidth consideration




If we find that we are running out of network bandwidth
then we will use just use ssh!
ssh -X will login and set the X Display for you.
If you have X11 on your desktop then ssh -X may be a
way to reduce bandwidth
Except for one or two lab exercises almost all can be
done with ssh
48
Demo

Lets see

Secure Global Desktop &

ssh
49
Lab Format


We have 5 tracks for this lab. You can pick one or more
1. DTrace basics
2. DTrace and JDK 7.0
3. DTrace for scripting languages
4. DTrace for system admins
5. DTrace advanced topics
Solutions to exercises are in

/export/home/dhol/solutions

All the docs are in http://dtracehol.com/

Lab docs are also available as

/export/home/dhol/dtracelab.pdf
50
Lab Introduction

Lab track explanation
51
Exercise - Overview
Each lab exercise is self contained.

Exercise 1 – 3
: DTrace introduction

Exercise 4
: DTrace Design Patterns

Exercise 5 – 6
: DTrace for Java 7.0 & 6.0

Exercise 7 – 10 : DTrace for the AMP stack

Exercise 11-12
: DTrace and Python/Ruby

Exercise 13
: DTrace toolkit

Exercise 14
: DTrace visualization

Exercise 15-16
: DTrace C & Java API

Exercise 17
: JDK 7.0 JSDT probes
52
Exercise – 5 Tracks
I.
DTrace fundamental Concepts
1. DTrace introduction and command line
2. D-Script syntax and examples
3. DTrace aggregates
4. DTrace Design Patterns
II. DTrace in Java 6.0 & 7.0 – Hotspot provider
5. Introduction to hotspot provider & GC probes
6. Hotspot – classloading, threads and methods
14. JSDT Embedding DTrace probes in Java applications.
III. DTrace for scripting languages and Web 2.0
7. DTrace and PHP
8. DTrace and Javascript
9. DTrace and MySQL
10. DTrace and AMP stack
11. DTrace and Python
12. DTrace and Ruby
53
Exercise – 5 Tracks
IV. DTrace for System Administrators
13.DTraceToolkit
14.Visualizing DTrace data - Chime
V. Advanced topics
15.libdtrace C API for DTrace
16.Java API for DTrace
17.JSDT Embedding DTrace probes in Java applications.
54
Introduction to reference slides

What you need to know to use
the next 200 slides?
55
Reference Slides


For your reference we have a few extra slides. These
are material about the various use of DTrace.
Here is how the slides are organized






Slide 63 - D-Language fundamentals
Slide 71 - Aggregates
Slide 78 - Variables in DTrace
Slide 83 - DTrace providers
Slide 86 - DTrace visualization - Chime
Slide 91 - DTrace toolkit
56
Reference Slides












slide 113 - Performance scenarios
slide 128 - DTrace & C
slide 136 - DTrace and distructive actions
slide 145 - DTrace and C++
slide 149 - DTrace and Java
slide 151 - DTrace and Java 6.0 & 7.0
slide 167 - DTrace and Java 1.4.2 & 1.5
slide 175 - DTrace and scripting languages
slide 177 - DTrace and PHP
slide 185 - DTrace and Javascript
slide 189 – DTrace and Python
slide 193 – DTrace and Ruby
57
Reference Slides

slide 198 - DTrace and Databases




slide 200 - DTrace and Postgres
slide 211 - DTrace and MySQL
slide 216 - DTrace and Oracleslide
slide 221 - Make your own DTrace provider

slide 227 - DTrace Providers examples

slide 246 - DTrace Data structures

slide 249 - Postmortem tracing

slide 252 - Speculative tracing

slide 255 - DTrace configuration

slide 259 - DTrace more resources.
58
Enjoy the Lab
59
Q&A
60
Angelo Rajadurai
[email protected]
Reference Slides
2008 JavaOneSM Conference – Lab 9400 DTrace for JDK 7.0 | Angelo Rajadurai | 62
D- Language
D-Language - Quick overview
D Language - Format.


probe description
/ predicate /
{
action statements
}
When a probe fires then action is executed if predicate evaluates true.
Example, “Print the pid and current working directory when bash forks”
sysinfo:::fork
sysfork.d
/execname==”bash”/
{
printf("bash(%d) forked. Current working directory %s\n",pid,cwd);
}
2008 JavaOneSM Conference – Lab 9400 DTrace for JDK 7.0 | Angelo Rajadurai |
64
Probe
Probes are points of instrumentation

Probes are made available by providers

Probes identify the module and function that
they instruments

Each probe has a name

These four attributes define a tuple that uniquely
identifies each probe
provider:module:function:name

Example
fbt:unix:mutex_init:entry
vminfo:genunix:swapin:pgswapin

65
Listing Probes.



Probes can be listed with the “-l” option to dtrace(1M)
 in a specific function with “-f function”
 in a specific module with “-m module”
 with a specific name with “-n name”
 from a specific provider with “-P provider”
Empty components match all possible probes.
Wild card can be used in naming probe.
66
Providers




Providers represent a methodology for instrumenting
the system
Providers make probes available to the DTrace
framework
DTrace informs providers when a probe is to be
enabled
Providers transfer control to DTrace when an enabled
probe is fired
Examples
syscall provider provides probes in every system call
fbt provider provides probe into every function in the kernel

67
Predicates



A predicate is a D expression
Actions will only be executed if the predicate
expression evaluates to true
A predicate takes the form “/expression/” and is
placed between the probe description and the
action
Examples
Print the pid of every process that is started by uidpred.d
0

proc:::exec-success
/uid == 0/
{
printf("New process %s(%d) started\n",execname, pid);
}
68
Actions





Actions are executed when a probe fires
Actions are completely programmable
Most actions record some specified state in the
system
Some actions change the state of the system system
in a well-defined manner
 These are called destructive actions and are
disabled by default.
Probes may provide parameters than can be used in
the actions.
69
Data types & Operators

D Language provides standard data types.

Integer types(32 bit size, 64-bit size)
 char(1,1), short(2,2),int(4,4), long(4,8),long long(8,8)

Float types(32-bit size, 64-bit size)
 float(4,4), double(8,8),long double(16,16)

Integer float and string constants are present in D.

add(+), sub(-), mul(*), div(/), mod(%) math operators

less than (<), less than or equal(<=) etc. defined

logical AND(&&), OR(||) & XOR(^^)

bitwise operations and standard assignment ops present.
70
Aggregation
Performance enhancer – huge time
saver!
Aggregation


Think of a case when you are want to know the “total” time the
system spends in a function.

We can save the amount of time spent by the function every time
it is called and then add the total.
 If the function was called 1000 times that is 1000 bits of info
stored in the buffer just for us to finally add to get the total.

Instead if we just keep a running total then it is just one piece of
info that is stored in the buffer.

We can use the same concept when we want averages, count,
min or max.
Aggregation is a D construct for this purpose.
72
Aggregation - Format
@name[keys] = aggfunc(args);

'@' - key to show that name is an aggregation.

keys – comma separated list of D expressions.

aggfunc could be one of...

sum(expr) – total value of specified expression

count()- number of times called.

avg(expr) – average of expression

min(expr)/max(expr) – min and max of expressions

quantize()/lquantize() - power of two & linear distribution
73
Aggregation Example 1.
#!/usr/bin/dtrace -s
sysinfo:::pswitch
{
@[execname] = count();
}
aggr.d
74
Aggregation Example 2.
aggr2.d
#!/usr/sbin/dtrace -s
pid$target:libc:malloc:entry
{
@["Malloc Distribution"]=quantize(arg0);
}
$ aggr2.d -c who
dtrace: script './aggr2.d' matched 1 probe
...
dtrace: pid 6906 has exited
Malloc Distribution
value ------------- Distribution ------------- ----------------------count
1|
2 |@@@@@@@@@@@@@@@@@
3
4|
8 |@@@@@@
16 |@@@@@@
1
32 |
`
64 |
128 |
256 |
512 |
1024 |
2048 |
4096 |
8192 |@@@@@@@@@@@
2
16384 |
0
0
1
0
0
0
0
0
0
0
0
0
75
More about aggregations





printa(@name) – print aggregation “name”

printa also takes a format string.
normalize() - normalize aggregation over time. (ie. average).

data is not lost just a view created
denormalize() - opposite of normalize

remove the normalized view
clear(@name) – clear and allow DTrace to reclaim aggregation mem.
trunc(@name, num) - truncate value and key after top num entries.

A negative num will truncate after bottom num entries.

Typically we are only interested in the top /bottom functions.
76
Aggregation example
syscall::mmap:entry
{
@a["number of mmaps"] = count();
@b["average size of mmaps"] = avg(arg1);
@c["size distribution"] = quantize(arg1);
}
profile:::tick-10sec
{
printa(@a);
printa(@b);
printa(@c);
clear(@a);
clear(@b);
clear(@b);
}
77
Variables
Calling all MT programmers!
Calculating time spent
One of the most common request is to find time spent in a given function.
Here is how this can be done – buggy code. See next slide for better
version
syscall::open*:entry,
syscall::close*:entry
{
ts=timestamp;
}
timespent.d
syscall::open*:return,
syscall::close*:return
{
timespent = timestamp - ts;
printf("ThreadID %d spent %d nsecs in %s", tid, timespent, probefunc);
ts=0; /*allow DTrace to reclaim the storage */
timespent = 0;
}
79
Thread Local Variable
self->variable = expression;

self – keyword to indicate that the variable is thread local

A boon for observers of multi-threaded apps..

As name indicates this is specific to the thread. See code re-written
{
self->ts=timestamp;
}
self.d
{
timespent = timestamp - self->ts;
printf("ThreadID %d spent %d nsecs in %s", tid, timespent, probefunc);
self->ts=0; /*allow DTrace to reclaim the storage */
timespent = 0;
}
80
Built-in Variable
Here are a few built-in variables.
arg0 ... arg9 – Arguments represented in int64_t format
args[ ] - Arguments represented in correct type based on
function.
cpu – current cpu id.
cwd – current working directory
errno – error code from last system call
gid, uid – real group id, user id
pid, ppid, tid – process id, parent proc id & thread id

probeprov, probemod, probefunc, probename - probe info.
timestamp, walltimestamp, vtimestamp – time stamp nano sec from an
arbitary point and nano sec from epoc.
81
External Variable

DTrace provides access to kernel & external variables.

To access value of external variable use `
dtrace:::BEGIN
{
printf("physmem is %d\n", `physmem);
printf("maxusers is %d\n", `maxusers);
printf("ufs:freebehind is %d\n", ufs`freebehind);
exit(0);
}


ext.d
Note: ufs`freebehind indicates kernel variable freebehind in the ufs
module.
These variables cannot be lvalue. They cannot be modified from
within a D Script
82
DTrace providers
What does Solaris provide for you!
Providers

Here is the list of providers

dtrace Provider – provider probes related to DTrace itself

lockstat Provider – lock contention probes

profile Provider – probe for firing at fixed intervals

fbt Provider – function boundary tracing provider

syscall Provider – probes at entry/return of every syscall

sdt Provider – “statically defined probes” user definable probe

sysinfo Provider – probe kernel stats for mpstat and sysinfo tools

vminfo Provider – probe for vm kernel stats

proc Provider – process/LWP creation and termination probes

sched Provider – probes for CPU scheduling
84
Providers - cont.

We will now see some more details on a few Solaris Providers

io Provider – provider probes related to disk IO

mib Provider – insight into network layer.

fpuinfo Provider – probe into kernel software FP processing

pid Provider – probe into any function or instruction in user code.

plockstat Provider – probes user level sync and lock code

fasttrap Provider – non-exposed probe. Fires when DTrace
probes are called.
85
DTrace Visualization
A sweet “chime”
Java API for DTrace

As you may have seen the output from DTace is always text.

If you are into GUI and graphs there is Java DTrace API and Chime.


Chime – is a graphical tool to visualize DTrace output.

The tool currently works only in Nevada Bld 35+

Will be available in Update 4.

You can visualize output from any D-script.
Java DTrace API is java binding for DTrace

Available in Nevada and Solaris 10 Update 4
87
Chime
Get it from
http://opensolaris.org/os/project/dtrace-chime/install/



Chime visualization is controlled by “display”
Displays are xml files that contains the D-scripts and the configuration
for the display.

A wizard is also provided for creating new “Displays”

This is a great tool for demo.

Speaking of demo's here is one.
88
jdtrace – Java version of dtrace
command

Get it from
http://opensolaris.org/os/project/dtrace-chime/java_dtrace_api/jdtrace/

This is a great example of the Java DTrace API.

Source provided.

Needs Chime installed as well as Nevada 50 and later.

Usage is similar to dtrace just add a j.

Example
# jdtrace -n syscall:::entry'{@[probefunc]=count()}'
89
DTrace - Real life.





We learnt about DTrace and how to develop D-scripts
Knowing to instrument is not the same as knowing where to
instrument.
Next we will see how to use DTrace in real life.
First we will see about the DTrace toolkit.

Over a hundred useful D-scripts

Become an expert in minutes.
Performance scenarios

A few real life scenarios and how to use Dtrace to address it.
90
DTrace Toolkit
How to become and instant hero!
DTrace Toolkit

The DTraceToolkit is a collection of useful

Documented scripts developed by the OpenSolaris DTrace
community

Available under www.opensolaris.org

Ready to use D scripts

The toolkit contains:
 the scripts
 the man pages
 the example documentation
 the notes files
92
DTrace Toolkit

Get it from
http://www.opensolaris.org/os/community/dtrace/dtracetoolkit

Download

gunzip & tar xvf

./install

A nice guide comes with the script

See the Docs/Contents for more details.

Set PATH to include /opt/DTT
93
DTrace Toolkit
Bin/
Symlinks to the scripts
Apps/
Application specific scripts
Cpu/
Scripts for CPU analysis
Disk/
Scripts for disk I/O analysis
Docs/
Contents
Examples/
Documentation
Command list for the Toolkit
Examples of command usage
Faq
Frequently asked questions
Links
Further DTrace links
Notes/
Notes on Toolkit commands
Readme
Extra/
Readme for using the docs
Misc scripts
Guide
This file!
Kernel/
Scripts for kernel analysis
License
The CDDL license
Locks/
Scripts for lock analysis
Man/
Man pages
man1m/
Mem/
Man pages for the Toolkit commands
Scripts for memory analysis
Net/
Scripts for network analysis
Proc/
Scripts for process analysis
System/
SM
JavaOne
Scripts for system2008
analysis
Conference – Lab 9400 DTrace for JDK 7.0 | Angelo Rajadurai |
94
DTrace Toolkit

Categories

Apps - scripts for certain applications: Apache, NFS

Cpu - scripts for measuring CPU activity

Disk - scripts to analyse I/O activity

Extra - other categories

Kernel - scripts to monitor kernel activity

Locks - scripts to analyse locks

Mem - scripts to analyse memory and virtual memory

Net - scripts to analyse activity of the network iinterfaces, and the
TCP/IP stack

Proc - scripts to analyse activity of a process

System - scripts to measure system wide activity

User - scripts to monitor activity by UID

Zones - scripts to monitor activity by zone
95
DTrace Toolkit

Apps





Used to measure and report certain metrics from applications like:
Apache Web server, NFS client, UNIX shell
httpdstat.d: computes real-time Apache web statistics: the number of
connections, GET, POST, HEAD and TRACE requests
nfswizard.d: used to measure the NFS client activity regarding
response time and file accesses
shellsnoop: captures keystrokes, used to debug and catch command
output. Use with caution !
weblatency.d: counts connection speed delays, DNS lookups, proxy
delays, and web server response time. Uses by default Mozilla
browser
96
DTrace Toolkit

CPU
 Reports and list the CPU activity like: cross calls,
interrupt activity by device, time spent servicing
interrupts, CPU saturation
 cputypes.d: lists the information about CPUs: the
number of physical install CPUs, clock
 loads.d: prints the load average, similar to uptime
 intbycpu.d: prints the number of interrupts by CPU
 intoncpu.d: lists the interrupt activity by device;
example: the time consumed by the ethernet driver,
or the audio device
 inttimes.d: reports the time spent servicing the
interrupt
97
DTrace Toolkit

CPU
 xcallsbypid.d: list the inter-processor cross-calls by
process id. The inter-process cross calls is an
indicator how much work a CPU sends to another
CPU
 dispqlen.d : dispatcher queue length by CPU,
measures the CPU saturation
 cpuwalk.d : identify if a process is running on
multiple CPUs concurrently or not
 runocc.d : prints the dispatcher run queue, a good
way to measure CPU saturation
98
DTrace Toolkit

Disk
 Analyses I/O activity using the io provider from
DTrace: disk I/O patterns, disk I/O activity by
process, the seek size of an I/O operation
 iotop: a top like utility which lists disk I/O events by
processes
 iosnoop: a disk I/O trace event application. The
utility will report UID, PID, filename regarding for a I/
O operation
 bitesize.d: analyze disk I/O size by process
 seeksize.d: analyzes the disk I/O seek size by
identifying what sort I/O operation the process is
making: sequential or random
99
DTrace Toolkit

Disk





iofile.d: prints the total I/O wait times. Used to debug
applications which are waiting for a disk file or resource
iopattern: computes the percentage of events that were of a
random or sequential nature. Used easily to identify the type of
an I/O operation and the average, totals numbers
iopending: prints a plot for the number of pending disk I/O
events. This utility tries to identify the "serialness" or
"parallelness" of the disk behavior
diskhits: prints the load average, similar to uptime
iofileb.d: prints a summary of requested disk activity by
pathname, providing totals of the I/O events in byte
100
DTrace Toolkit

FS





Analyses the activity on the file system level: write
cache miss, read file I/O statistics, system calls
read/write
vopstat: traces the vnode activity
rfsio.d: provides statistics on the number of reads:
the bytes read from file systems (logical reads) and
the number of bytes read from physical disk
fspaging.d: used to examine the behavior of each I/
O layer, from the syscall interface to what the disk is
doing
rfileio.d: similar with rfsio.d but reports by file
101
DTrace Toolkit

Kernel
 Analyses kernel activity: DNLC statistics, CPU time
consumed by kernel, the threads scheduling class
and priority
 dnlcstat: inspector of the Directory Name Lookup
Cache (DNLC)
 cputimes: print CPU time consumed by the kernel,
processes or idle
 cpudist: print CPU time distributions by kernel,
processes or idle cswstat.d: prints the context
switch count and average
 modcalls.d: an aggregation for kernel function calls
by module
102
DTrace Toolkit

Kernel
 dnlcps.d: prints DNLC statistics by process
 dnlcsnoop.d: snoops DNLC activity
 kstat_types.d: traces kstat reads
 pridist.d: outputs the process priority distribution.
Plots which process is on the CPUs, and under
what priority it is
 priclass.d: outputs the priority distribution by
scheduling class. Plots a distribution
 whatexec.d: determines the types of files which are
executed by inspected the first four bytes of the
executed file
103
DTrace Toolkit

Lock
 Analyses lock activity using lockstat provider
 lockbydist.d: lock distribution by process name
 lockbyproc.d: lock time by process name
104
DTrace Toolkit

Memory
 This category analyses memory and virtual memory
things: virtual memory statistics, page management,
minor faults
 vmstat.d: a vmstat like utility written in D
 vmstat-p.d: a vmstat like utility written in D which
does display what “vmstat -p” does: reporting the
paging information
 xvmstat: a much improved version of vmstat which
does count the following numbers: free RAM, virtual
memory free, major faults, minor faults, scan rate
105
DTrace Toolkit

Memory
 swapinfo.d: prints virtual memory info, listing all
memory consumers related with virtual memory
including the swap physical devices
 pgpginbypid.d: prints information about pages
paged in by process id
 minfbypid.d: detects the biggest memory consumer
using minor faults, an indication of memory
consumption
106
DTrace Toolkit

Network





These scripts analyse the activity of the network interfaces and
the TCP/IP stack. Some scripts are using the mib provider.
Used to monitor incoming
icmpstat.d: reports ICMP statistics per second, based on mib
tcpstat.d: prints TCP statistics every second, retrieved from
the mib provider: TCP bytes received and sent, TCP bytes
retransmitted
udpstat.d: prints UDP statistics every second, retrieved from
the mib provider
tcpsnoop.d: analyses TCP network packets and prints the
responsible PID and UID. Useful to detect which processes
are causing TCP traffic.
107
DTrace Toolkit

Network
 connections: prints the inbound TCP connections.
This displays the PID and command name of the
processes accepting connections
 tcptop: display top TCP network packets by
process. It can help identify which processes are
causing TCP traffic
 tcpwdist.d: measures the size of writes from
applications to the TCP level. It can help identify
which process is creating network traffic
108
DTrace Toolkit

Process







Analyses process activity: system calls/process, bytes written
or read by process, files opened by process,
sampleproc: inspect how much CPU the application is using
threaded.d: see how well a multithreaded application uses its
threads
writebytes.d: how many bytes are written by process
readbytes.d: how many bytes are read by process
kill.d: a kill inspector. What how signals are send to what
applications
newproc.d: snoop new processes as they are executed.
109
DTrace Toolkit

Process






syscallbyproc.d & syscallbypid.d: system calls by process
or by PID
filebyproc.d: files opened by process
fddist: a file descriptor reporter, used to print distributions for
read and write events by file descriptor, by process. Used to
determine which file descriptor a process is doing the most I/O
with
pathopens.d: prints a count of the number of times files have
been successfully opened
rwbypid.d: reports the no. of read/writes calls by PID
rwbytype.d: identifies the vnode type of read/write activity whether that is for regular files, sockets, character special
devices
110
DTrace Toolkit

Process






sigdist.d: prints the number of signals received by process
and the signal number
topsysproc: a report utility listing top number of system calls
by process
pfilestat: prints I/O statistics for each file descriptor within a
process. Very useful for debug certain processes
stacksize.d: measures the stack size for running threads
crash.d: reports about crashed applications. Useful to identify
the last seconds of a crashed application
shortlived.d: snoops the short life activity of some processes
111
DTrace Toolkit

System





Used to measure system wide activity
uname-a.d: simulates 'uname -a' in D
syscallbysysc.d: reports a total on the number od system
calls on the system
sar-c.d: reports system calls usage similar to 'sar -c'
topsyscall: prints a report of the top system calls on the
system
112
DTrace Performance
Scenarios
DTrace for those with a day job!
DTrace for every day use.




We now move on how to apply DTrace for your day job.
We look at some broad guide lines and then talk about some more
detailed scenarios.
The goal of this section is to provide you with some ideas on how
to use DTrace.
I have heavily borrowed from Benoit Chaffanjon's presentation for
this section – Thanks.
An Ounce of Data is worth more than many
pounds of opinions.
114
Supplement not Replace


One of the first thing to understand is that DTrace supplements
the Solaris tools and does not indent to replace it.
Continue to use tools like

truss gives you very quick access to syscalls for a given
process with some level of aggregation. But watch out for
overhead.

vmstat mpstat iostat almost all real life scenarios will start
with using these tools. They are the pillars of performance
tuning. Use them to get overview and then use dtrace to dig
down deep.

/proc tools – extremely useful to get quick overview of
processes.

trapstat cpustat sputrack intrstat – Harware statistics.
115
Scenarios

We will look at the following scenarios
1. High user time
2. High system time
3. Multi-threaded applications
4. High cross calls
5. Network traffic
6. Disk usage
116
1. High User Time

How do you find that you have high user time

Use mpstat vmstat or iostat


If the “User” column is very high then you can use this scenario.
Look at prstat to see if there is one app or a set of apps that are on the
top.

If it is then you can use the following to see what that app is doing
# dtrace -n profile-1001'/pid==1234/{@[ustack(1)]=count()}'
# dtrace -n profile-1001/execname==”app”/'{@[ustack(1)]=count()}'

If not then try to see if there is a pattern
# dtrace -n profile-1001/arg1/'{@[ustack(1)]=count()}'
arg1 – pc if the CPU is running user land code.
117
1. High User Time

If the output looks like
libm.so.2èxp+0x4
94
libm.so.2èxp+0x54
99
libm.so.2èxp+0x64
111
libm.so.2èxp+0x98
116
multiìntio_calc_n_month_rate+0x8f
117
multiìntio_calc_n_month_rate+0x186
138
libm.so.2èxp+0x10d
163
libm.so.2èxp+0x90
184
libm.so.2èxp+0x61
617
libm.so.2`log+0x1a
854
118
1. High User Time


Looks like you are in a lot of libm apps (exp, log)

Make sure you are using the best of these functions

Look at possibly inlining.

Also you may want to look at the application logic to see if this
can be mitigated.
You can also use the following script to see details of each call.

Beware it has significant probe effect.
#!/usr/sbin/dtrace -FS
pid$1:::entry, pid$1:::return
{
trace(timestamp);
}
Or use the dapptrace DTraceToolkit
119
1. High User Time
Other things to watch for

Calls to .mul(), .dev() etc on SPARC
App may have been compiled for v7 – recompile

Excessive getenv(), getrusage() getrlimit()
Look at application logic to see

Excessive time(), gettimeofday()
Replace with gethrtime()

Watch for excessive memmove
Use memcpy() if regions do not overlap

120
1. High User Time

Looks for where mem* apps are coming from
dtrace -n pid1234::mem*:entry'{@[probemod,probefunc]=count()}'





This will print the function and lib name.
Look for significant FPU usage.

kstat -n fpu_traps
Consider AMD
If high FPU usage on T2000s then beware

Use cooltst to see if the workload fits CMT.
If you do not see one or one class of applications showing up in
the top then look at execsnoop from the toolkit to see if there are
a bunch of jobs being started.
121
2. High System Time / Calls


High system Calls

Look at the vmstat under the Sys Faults
Use the following script to get a quick idea who is causing it.
# dtrace -n syscall:::entry'{@[execname]=count()}'

If you want to look at more details use
# dtrace -n syscall:::entry'{@[execname,probefunc]=count()}'

Use the argN for more details of particular system call.

Example to find what file is being read()/write()/send()/recv() etc
# dtrace -n syscall:::read:entry'{ printf("%s(%d) read file %s\n",
execname, pid, fds[arg0].fi_pathname) }'

Look at topsysproc from the toolkit.
122
2. High System Time / Calls

How to recognize high system time

Look at vmstat under sys time

How to use DTrace

Following scripts will tell you what execname(pid) is using kernel
# dtrace -n profile-1001'/arg0/{@[execname,pid]=count()}'

If you want to dig deeper and look at where in the kernel you are
# dtrace -n profile-1001'/arg0/{@[stack(1)]=count()}'

If you suspect locking use lockstat to drill down
123
3. Multi-threaded apps




Tools like truss are not enough to analyze threaded apps.
Start with prstat -mL 1

Look at the Lck section to see if there is high amount of
locking

If so use ploctstat (a dtrace consumer) to better understand.
Pay close attention to locks taken by malloc.

libc malloc does not scale use libmtmalloc (LD_PRE_LOAD)
If there is a huge use of mutex_locks for small segments of code
just to increment a value replace with atomic ops see atomic_ops
(3c)
124
4. High Cross Calls

How to recognize high cross calls

Use mpstat and see xcalls section

You can use xcals probe to find who is making these calls.

Look at xcallsbypid.d in the toolkit

Look at the ustack() when making xcalls.

Some usual culprits are segmap – consider Direct IO

Also look for unneeded munmap/mmaps
125
5. Network traffic



The network status utility netstat displays a status of all network
connections on a system. With the current tools there is no easy
way to find out and co-relate a network connection with a process
or the owner of it
Extra tools like lsof can list what connections were made and by
who What about incoming connections ?
Solve the problem by using: tcptop, tcpsnoop and connections
utilities from DTT

tcpsnoop – similar to snoop but just for tcp and light weight

tcptop – finally a top for tcp traffic

connections – watch the connections being made in the
system
126
6. Disk I/O







Disk utilisation can be monitored using iostat but to co-relate the
utilisation with a process is a hard mission
There are tools to check CPU usage by process but there are no
tools to check disk I/O by process
The old good friend: iostat -xnmp I/O type: reading iostat data a
SysAdmin can not describe if the I/O is sequential or random
It is important to know what type of I/O there is: sequential or
random
How can you list what processes are generating I/O, or list disk
events or how much a process is using the disk (size of the disk
event or the service time of the disk events) ?
Easily use the following DTT scripts: iotop, iosnoop from DTT
root directory. Look at bitsize.d, seeksize.d iopattern
Try dtrace -n io:::start'{printf(“%s(%d) \n”, execname, pid,
args[0]->b_count)}'
127
DTrace for C Apps
For thos who debate K&R or ANCII
The pid Provider


The pid Provider is extremely flexible and allowing you to
instrument any instruction in user land including entry and exit.
pid provider creates probes on the fly when they are needed. This
is why they do not appear in the dtrace -l listing
We will see how to use the pid provider to trace
Function Boundaries
Any arbitrary instruction in a given function

129
pid – Function Boundary probes
The probe is constructed using the following format
pid<processid>:<library or executable>:<function>:<entry or return>

Examples:
pid1234:date:main:entry
pid1122:libc:open:return


Following code counts all libc calls made by a program
pid$target:libc::entry
{
@[probefunc]=count()
}
pid1.d
130
pid – Function Offset probes
The probe is constructed using the following format
pid<processid>:<library or executable>:<function>:<offset>

Examples:
pid1234:date:main:16
pid1122:libc:open:4


Following code prints all instructions executed in the programs main
pid$target:a.out:main:
{
offs.d
}
131
DTrace User Process




DTrace provides a lot of features to probe into the user
process
We will look at routines that are specific to user code
tracing.
Some examples of using DTrace in user code will be
discussed
We will talk about “modifying” user process state
 A real neat example is discussed
 This could be used in fault injection
132
Action & Subroutines
There are a few actions and subroutines that are useful for user land
dtrace.

ustack(<int nframes>, <int strsize>) - records user process stack.
nframes – specifies the number of frames to record
strsize – if this is specified and non 0 then the address to name
translation is done when the stack is recorded into a buffer of
strsize. This will avoid problem with address to name
translation in user land when the process may have exited

For java code analysis you'd need Java 1.5 to use this ustack() functionality
(see example)
133
copyin & copyinstr


DTrace executes in the kernel context
As user process run in the user address space you'd need some
special handling to get the data from the user address space to the
kernel address space.
The copyin & copystr subroutines help you do this data copying.
copyin(addr,len) – Copy 'len' bytes of data from addr in user land to
kernel space and return address.
copyinstr(addr) – Copy string from addr in userland to kernel and return a
string.

134
copyin & copyinstr example.
!/usr/sbin/dtrace -s
syscall::write:entry
/pid == $target/
{
printf("Write %s\n",copyinstr(arg1)); /*elegant way*/
}
cpy.d
Signature of write system call
ssize_t write(int fd, const void *buf, size_t nbytes);
$ ./cpy.d -c date
dtrace: script './cpy.d' matched 1 probe
Fri Jan 28 13:40:51 PST 2005
dtrace: pid 4342 has exited
CPU ID
FUNCTION:NAME
0 13
write:entry Write Fri Jan 28 13:40:51 PST 2005
135
DTrace Destructive actions
How to use DTrace for more than
just observation
Destructive actions



We saw that DTrace is safe to use on a live system because there are
checks to make sure it does not modify what it observes.
There are some cases where you want to change the state of the
system.
Example

stop a process to better analyze it.

kill a runaway process

run a process using system

These actions are disabled by default.

Use '-w' option to enable it.

We will see some of these destructive actions.
137
Destructive actions





stop() - stop the process that fired the probe. Use prun to make the
process continue. Use gcore to capture a core of the process.
raise(int signal) – Send signal to the process that fired the probe.
copyout(void *buf, uintptr_t addr, size_t nbytes) – copy nbytes from
address pointed by buf to addr in the current proc's user address
space.
copyoutstr(string str, uintptr_t addr, size_t maxlen) – copy string (null
terminated) from str to addr in the current proc's user address space.
Max length is maxlen.
system(string program) – Similar to system call in C. Run the
program. system also allows you to use printf formats in the string.
138
Porting Solaris to PPC – DTrace style
#!/usr/sbin/dtrace -Cs
#include<sys/systeminfo.h>
#pragma D option destructive
port.d
syscall::uname:entry
{
self->addr = arg0;
}
syscall::uname:return
{
copyoutstr("SunOS", self->addr, 257);
copyoutstr("PowerPC", self->addr+257, 257);
copyoutstr("5.10", self->addr+(257*2), 257);
copyoutstr("Generic", self->addr+(257*3), 257);
copyoutstr("PPC", self->addr+(257*4), 257);
}
139
Porting Solaris to PPC – Dtrace style
syscall::systeminfo:entry
/arg0==SI_ARCHITECTURE/
{
self->arch = arg1;
}
syscall::systeminfo:return
/self->arch/
{
copyoutstr("PowerPC", self->arch,257);
self->arch=0;
}
/arg0==SI_PLATFORM/
{
self->mach = arg1;
}
/self->mach/
{
copyoutstr("APPL,PowerBook-G4", self->mach,257);
self->mach=0;
}
140
Containers – DTrace style
#!/usr/sbin/dtrace -Cs
#include<sys/systeminfo.h>
contain.d
#pragma D option destructive
syscall::uname:entry
/ppid == $1/
{
self->addr = arg0;
}
syscall::uname:return
/ppid == $1/
{
copyoutstr("SunOS", self->addr, 257);
copyoutstr("PowerPC", self->addr+257, 257);
copyoutstr("5.10", self->addr+(257*2), 257);
copyoutstr("Generic", self->addr+(257*3), 257);
copyoutstr("PPC", self->addr+(257*4), 257);
}
141
Containers – Dtrace style
/arg0==SI_ARCHITECTURE/
{
self->arch = arg1;
}
/self->arch && ppid == $1/
{
copyoutstr("PowerPC", self->arch,257);
self->arch=0;
}
/arg0==SI_PLATFORM && ppid == $1/
{
self->mach = arg1;
}
/self->mach && ppid == $1/
{
copyoutstr("APPL,PowerBook-G4", self->mach,257);
self->mach=0;
}
142
Who killed the process?
proc:::signal-send
/args[1]->pr_fname == $$1/
{
printf("%s(pid:%d) is sending signal %d to %s\n", execname, pid, args[2],args[1]->pr_fname);
}
sig1.d
$ ./sig1.d bc
sched(pid:0) is sending signal 24 to bc
sched(pid:0) is sending signal 24 to bc
bash(pid:3987) is sending signal 15 to bc
The above program prints out process that is sending the
signal to the program “bc”.
Note: $$1 is argument 1 as string
The signal-send probe has arg1 that has info on signal
destination
The signal-send probe has args2 that has the signal number
143
Stop that killer.
#!/usr/sbin/dtrace -wqs
proc:::signal-send
/args[1]->pr_fname == $$1/
{
printf("%s(pid:%d) is sending signal %d to %s\n", execname, pid, args[2],args[1]->pr_fname);
stop();
}
sig2.d
$ ./sig2.d bc
This one is modified to include destructive actions (-w)
We do the stop() which stops the sending process.
You can then examine the sending process.
Note: We are not stoping the signal delivery itself.
144
DTrace for C++
From Brian Cantrill to Bjarne
Stroustrup!
C++ Name Mangling

The C++ compiler mangles symbols

Sun's mangling rules are different from the g++ rules


DTrace cannot demangle the symbols as it does not make an
assumption on the compiler.
So we need to demangle the symbols for dtrace.
Fortunately due to Unix magic this is pretty simple
dem `nm execname | cut -d'|' -f 8`
(OK its not magic. Just do a nm (print symbol list) of an execname. Cut
only the 8th field and use the result as the input to the demagler. If this
sounds complicated – Just do it.)

146
C++ & DTrace example

We will use a simple example of using DTrace with C++.

A star of an example - staroffice8

Looking at live running staroffice app.

Find the pid of staroffice. “pgrep -l staroffice” works well.

Use oneliner
dtrace -n pid`pgrep soffice.bin`:::entry'{ @[probefunc]=count() }' -o
soffice.out

Stop script after a few minutes to collect info.

Now run the output through c++filt

DEMO
147
C++ & DTrace example

On my system void*operator new(unsigned) = __1c2n6FI_pv_

Dtrace one liner for
dtrace -n pid`pgrep
soffice.bin`::__1c2n6FI_pv_:entry'{ @=quantize(arg0) }'
value ------------- Distribution ------------- count
-1 |
0
0|
7
1|
54
2|
0
4 |@@
9726
8 |@@@@
20913
16 |@@@@@@@@@@@@@@
32 |@@@@@@@
37191
64 |@@@@@@@@
128 |@@
256 |@@
73216
42712
10412
11134
148
DTrace and Java
Welcome to the Duke era!
DTrace and Java


We already saw how to use ustack() on a java process.
We will see how dtrace can shed more light on...

When VM starts and ends

When thread starts and ends

When class gets loaded and unloaded

When object allocated and freed

When GC starts and ends

When a method is called and method returns
150
DTrace for Java 6.0
Tracing Java on a Mustang!
DTrace and Mustang

JDK 6.0 (Mustang) has some very interesting embedded DTrace
probes. Mustang provides probes in the following areas.

VM lifecycle

Thread lifecycle

Garbage collection

Classloading

Method compilation

Monitor probes

Method entry and exit probes

JNI probes
152
DTrace and Mustang


We will now look into details on some of these areas.
For each of these areas we will look at

Probe names

Arguments provided

Sample scripts
153
DTrace and Mustang - args

For most of these probes the arguments have the following
format.

First argument gives string data

The second one gives the size
So to get to the data you'd use the following format
str_ptr = (char*) copyin(arg0, arg1+1);
str_ptr[arg1] = '\0';
name = (string) self->str_ptr;


As the arguments are not null terminated we cannot use copyinstr
()

The above code segment add the null termination to the string.

In Solaris.next you can use copyinstr(arg0,arg1).
154
DTrace and Mustang – VM life cycle
probes
Probe names
vm-init-begin – Probe fires when JVM initialization starts
vm-init-end – Probe fires when JVM initialization ends and JVM is
ready to run user code
vm-shutdown – Probe fires when JVM shuts down.

155
DTrace and Mustang – Thread life
cycle
Probe names
thread-start – Probe fires when thread start
thread-stop – Probe fires when thread completed

Arguments
arg0 – String with thread name
arg1 – Length of the thread name string
arg2 – Thread id
arg3 – Solaris Thread id.
arg4 – Boolean. Is the thread a daemon or not.

156
DTrace and Mustang – Thread life
cycle
Sample code.
hotspot$1:::thread-start
{
self->ptr = (char *)copyin(arg0,arg1+1);
self->ptr[arg1]='\0';
self->tname=stringof(self->ptr);

printf("Thread %s started with Java TID %d Solaris TID
%d",self->tname,arg2,arg3);
}
Provide pid of Java process as first argument
157
DTrace and Mustang – Garbage
Collector probes
Probe names
gc-begin – Probe fires when gc start
gc-end – Probe fires when gc completed
mem-pool-gc-begin – GC for individual pool starts
mem-pool-gc-end - GC for individual pool ends

Arguments
arg0,arg1 – name and length of memory pool manager
arg2,arg3 – name and length of memory
arg4 – initial memory pool size in bytes
arg5 – memory in use in the pool.
arg6 – number of committed pages
arg7 – max size of memory pool

158
DTrace and Mustang – Garbage
collector probes
hotspot$1:::gc-begin
{
printf("GC Started at %Y\n",walltimestamp);
self->ts=timestamp;
}
hotspot$1:::gc-end
/self->ts/
{
printf("\t and ran for %d ms\n",(timestamp-self->ts)/1000);
}
Provide pid of java process as first argument.
159
DTrace and Mustang – Classloading
probes
Probe names
class-loaded – Probe fires when a new class is loaded
class-unloaded – Probe fires when a class is unloaded

Arguments
arg0,arg1 – name and length of class that is loaded
arg2 – id of the class loader
arg3 – boolean. Is this class a shared class

160
DTrace and Mustang – classloader
probes
hotspot$1:::class-loaded
{
self->cname=stringof(self->ptr);
printf("%s loaded\n",self->cname);
}
161
DTrace and Mustang – method calls
Probe names
method-entry – Probe fires when a new method is called.
method-return – Probe fires when method returns

Arguments
arg0 – java threadid
arg1,arg2 – name and length of the class of the method
arg3,arg4 – name and length of the method name.
arg5, arg6 – method signature string and length


Note: For these probes to be available the JVM needs to be
started with -XX:+ExtendedDTraceProbes
162
DTrace and Mustang – method calls
hotspot$1:::method-entry
{
self->str_ptr = (char*) copyin(arg1, args[2]+1);
self->str_ptr[args[2]] = '\0';
self->class_name = stringof(self->str_ptr);
self->str_ptr = (char*) copyin(arg3, args[4]+1);
self->str_ptr[args[4]] = '\0';
self->method = stringof(self->str_ptr);
}
@functions[self->class_name,self->method]=count();
JVM needs to be started with -XX:+ExtendedDTraceProbes
163
DTrace and Mustang – object
allocation
Probe names
object-alloc – Probe fires when a new method is called.
method-return – Probe fires when method returns

Arguments
arg0 – java thread id
arg1,arg2 – name and length of the class of the object allocated
arg3– size of object being allocated


Note: For these probes to be available the JVM needs to be
started with -XX:+ExtendedDTraceProbes
164
DTrace and Mustang – object
allocation
hotspot$1:::object-alloc
{
self->cname = stringof(self->ptr);
printf("Object of class %s allocated: Size %d\n",self->cname,
arg3);
}
JVM needs to be started with -XX:+ExtendedDTraceProbes
165
DTrace and Mustang 
The following probes are built into Mustang

When VM starts and ends

When thread starts and ends

When class gets loaded and unloaded

When object allocated and freed

When GC starts and ends

When a method is called and method returns

JNI calls
166
DTrace for Java
1.4.2 & 1.5
Well moving at Mustang speed is not for all.
dvm provider

Next version of java (6.0 – mustang) will support dtrace.

java.net project to add dtrace support in 1.4.2 and 1.5
https://solaris10-dtrace-vm-agents.dev.java.net/

Download shared libs

libdvmti.so – java 1.5

libdvmpi.so – java 1.4.2

Add location of libs to LD_LIBRARY_PATH variable

Set JAVA_TOOL_OPTIONS to -Xrundvmti:all

Name of provider - “dvm”
168
dvm provider




Some performance considerations
Dvm has considerable performance impact even when probes are
not turned on.
Setting JAVA_TOOL_OPTIONS to -Xrundvmti:all

Heavy performance impact
Use the dynamic option

JAVA_TOOL_OPTIONS=-Xrundvmti:all,dynamic=pipe_name

Write one char to pipe to turn probes on and off.

Considerable performance improvements (5x)
169
dvm provider – GC times

To see time taken by GC.
java_gc.d
dvm$target:::gc-start
{
self->ts = vtimestamp;
}
dvm$target:::gc-finish
{
printf("GC ran for %d nsec\n", vtimestamp - self->ts);
}
# ./java_gc.d -p `pgrep -n java`
170
dvm provider - alloc/free

To see objects that are allocated and freed.
java_alloc.d
dvm$target:::object-alloc
{
printf("%s allocated %d size object\n",copyinstr(arg0), arg1);
}
dvm$target:::object-free
{
printf("%s freed %d size object\n",copyinstr(arg0), arg1);
}
# ./java_alloc.d -p `pgrep -n java`
171
dvm provider - methods

To get count of java methods called.
java_method_count.d
dvm$target:::method-entry
{
@[copyinstr(arg0),copyinstr(arg1)] = count();
}
# ./java_method_count.d -p `pgrep -n java`
172
dvm provider - time-spent

To get time-spent on java methods.
dvm$target:::method-entry
java_method.d
{
self->ts[copyinstr(arg0),copyinstr(arg1)] = vtimestamp;
}
dvm$target:::method-return
{
@ts[copyinstr(arg0),copyinstr(arg1)] = sum(vtimestamp self->ts[copyinstr(arg0),copyinstr(arg1)]);
}
# ./java_method.d -p `pgrep -n java`
173
dvm provider - probes

Here is a list of dvm probes and their signatures













vm-init()
vm-death();
thread-start(char *thread_name);
thread-end();
class-load(char *class_name);
class-unload(char *class_name);
gc-start();
gc-finish();
gc-stats(long used_objects, long used_object_space);
object-alloc(char *class_name, long size);
object-free(char *class_name);
method-entry(char *class_name, char *method_name, char *method_signature);
method__return(char *class_name, char *method_name, char *method_signature);
174
DTrace for scripting
languages
Dynamic Tracing for Dynamic development
DTrace and Scripting Language


DTrace has been integrated into many scripting languages.
We will see

PHP

JavaScript

Python

Ruby
You can get all the tools from the coolstack
http://cooltools.sunsource.net/coolstack/index.html

176
DTrace for PHP
D tracing for P Hyper Processor
DTrace and PHP

DTrace provider for PHP is a PECL/PEAR module
To get the provider
# pear install dtrace
or
# pecl install dtrace
then add
“extension=dtrace.so” to the php.ini file
then
restart PHP

You can get these all pre-built from coolstack as well
http://cooltools.sunsource.net/coolstack/index.html

178
DTrace and PHP

Couple links for more information

Wez Furlong who created the provider has details in
http://netevil.org/node.php?
uuid=42f82a8b-09b7-5013-1667-2f82a8b24ead

Bryan Cantrill has a nice demo script in his blog.
http://blogs.sun.com/bmc/entry/dtrace_and_php_demonstrated
179
DTrace and PHP
There are two probes in the provider.
function-return
function-entry


You can use the 5 args in the action.

arg0 = the function name

arg1 = the filename

arg2 = the line number

arg3 = classname (or an empty string)

arg4 = object/class operator (::, ->, or an empty string)
180
DTrace and PHP

Here is a PHP example.
prime.php
<?php
$numprimes=10000; $pArray[0]=3; $test=5; $num=0;
function doesDevide($x,$y) { return($x % $y); }
function isprime($x)
{
global $pArray; global $num; $index=0; $check=1;
while($check==1 && $index <= $num &&
$x >= ($pArray[$index] * $pArray[$index]) )
{
if( doesDevide($x , $pArray[$index]) == 0) { $check=0; }
else $index++;
}
return($check);
}
181
DTrace and PHP

Here is a PHP example(cont)
while($num<$numprimes)
{
if(isPrime($test)==1){
$num++; $pArray[$num]=$test;
if($num%1000==0){
printf("Progress done ...%d\n",$num);
}
}
$test+=2;
}
?>
182
DTrace and PHP

Here is a simple D-Script
#!/usr/sbin/dtrace -Zqs
php*:::function-entry
{
@[copyinstr(arg0)]=count();
}


Note: -Z will allow probes that have zero match.
Run this script in one window while you run the php script in
another window.
183
DTrace and PHP
Example run

./php.d
^C
printf
10
isprime
52378
doesDevide
684216
184
DTrace for
Javascript
Not just for Java – but for Javascript as well
DTrace and Javascript


DTrace probes have been added to Mozilla to help observe Javascript
application.
You can download firefox with probes build in from

http://opensolaris.org/os/project/mozilla-dtrace

DTrace toolkit has a set of script that you can use for javascript

We will now look at the list of probes and a few examples
186

Here are the probes in the javascript provider
function-entry & function return – entry and return of javascript funcs
function-info & function-args – function info and arguments
object-create – object creation
object-create-start & object-create-done – finer info on obj creation
layout-start & layout-end – Start and end of layout
187
Here is an example that prints java script function flow.
#!/usr/sbin/dtrace -ZFs
javascript*:::js_function-entry
{
trace(copyinstr(arg2));
}

javascript*:::js_function-return
{
trace(copyinstr(arg2));
}
188
DTrace for Python
How to snake your way through python code
DTrace and Python

DTrace probes have been added to python

With these DTrace probes you can observe python script


One of the special features is that ustack() works well on these
probes
These probes are built into OpenSolaris build 65 and later. See

http://blogs.sun.com/levon/entry/python_and_dtrace_in_build

DTrace toolkit has a set of script that you can use for Python

We will now look at the list of probes and an example
190
DTrace and Python -probes

Here are the probes that are provided in the python provider

function-entry – fires when you call a function

function-return – fires when you return from function
 Both these provide the following args
arg1 – filename
arg2 – subroutine name
arg3 – line number
191
DTrace and Python - example

Here is a simple script to view python function calls
This one will print a summary of all the python function calls
#!/usr/sbin/dtrace -Zs
python*:::function-entry
{
}
192
DTrace for Ruby
A gem of a find!
DTrace and Ruby

DTrace probes have been added to Ruby by Joyeur

With these DTrace probes you can observe python script
Some good info on Ruby and DTrace can be found on the Joyeur site
http://www.joyeur.com/2007/05/07/dtrace-for-ruby-is-available


You can get the probe from the above site and some good examples

DTrace toolkit has a set of script that you can use for ruby

We will now look at the list of probes and examples
194
DTrace and Python -probes

Here are the probes that are provided in the ruby provider
195
DTrace and Python -probe arguments

Ruby probes provide some arguments as well
196
DTrace and Ruby - example

Here is a simple script to view Ruby function calls
This one will print a summary of all the ruby function calls
#!/usr/sbin/dtrace -Zs
ruby*:::function-entry
{
}
197
DTrace for
Databases
DTrace -n for select *
DTrace and Databases




DTrace has a lot of great features specially for databases
Look at the IO provider.

This will give you some ideas on which disk is being used
There are also some more specific help DTrace can provide for some
databases.
We will see three examples

PostgreSQL – embedded probes

MySQL – couple examples of DTrace usage.

Oracle – printing SQL statements
199
DTrace & Postgres
Observe post deploy of PostgreSQL
DTrace and PostgreSQL

Postgres 8.2 and later has following embedded D probes.

probe transaction__start(int);

probe transaction__commit(int);

probe transaction__abort(int);

probe lwlock__acquire(int, int);

probe lwlock__release(int);

probe lwlock__startwait(int, int);

probe lwlock__endwait(int, int);

probe lwlock__condacquire(int, int);

probe lwlock__condacquire__fail(int, int);

probe lock__startwait(int, int);

probe lock__endwait(int, int);
201
To get the probes

Download the source

For 32-bit version
./configure –enable-dtrace

For 64 bit:
$ configure CC='gcc -m64' –enable-dtrace \ DTRACEFLAGS='-64'

$ configure CC='/opt/SUNWspro/bin/cc -xtarget=native64'\ --enabledtrace DTRACEFLAGS='-64'

Run make or gmake.
202

We will use pgbench to look at the PostgresSQL probes.
Here are the steps for setup

Download an build postgres with D probes see ealier slide

Create user postgres
# useradd -c PostgreSQL -d /export/home/postgres -m -g other s /bin/bash postgres

Switch user to postgres
# su – postgres

Create Postgres database
$ initdb -D /export/home/postgres/pgdata

203
Start the database
$ createdb bench

Populate the data in db
$ pgbench -i -s 5 bench

Run the benchmark
$ pgbench -c 2 -t 400000 bench

Find the pids of the postgres process.
$ pgrep -l postgres

204

Here are a few example D-scripts.
postgres_avg_query_time.d
#!/usr/sbin/dtrace -Zqs
postgresql*:::transaction-start
{
self->ts=timestamp;
@cnt[pid]=count();
}
postgresql*:::transaction-commit
{
@avg[pid]=avg(timestamp - self->ts);
}
205
tick-5sec
{
normalize(@avg, 1000000);
printf("%15s %30s %30s\n","PID","Total queries","Avegrage time (ms)");
printf
("\t================================================================
======\n");
printa("%15d %@30d %@30d\n",@cnt,@avg);
printf
("\t================================================================
======\n\n");
clear(@cnt);
clear(@avg);
}
206
# ./postgres_avg_query_time.d
PID
Total queries
Avegrage time (ms)
==================================================================
23814
46
57
23817
58
34
23816
59
32
23815
59
33
23818
75
26
==================================================================
207
Here is a simple script to print all the SQL statements executed.
postgres_queries.d

#!/usr/sbin/dtrace -Zwqs
BEGIN
{
freopen(“sql.output”);
}
pid$1::pg_parse_query:entry
{
printf("%s\n",copyinstr(arg0));
}

This script will send its output to a file. sql.output
208

Sample output.
BEGIN;
UPDATE accounts SET abalance = abalance + 344 WHERE aid = 212898;
SELECT abalance FROM accounts WHERE aid = 212898;
UPDATE tellers SET tbalance = tbalance + 344 WHERE tid = 22;
UPDATE branches SET bbalance = bbalance + 344 WHERE bid = 3;
INSERT INTO history (tid, bid, aid, delta, mtime) VALUES (22, 3, 212898, 344, CURRENT_TIMESTAMP);
END;
BEGIN;
UPDATE accounts SET abalance = abalance + 15549 WHERE aid = 474266;
SELECT abalance FROM accounts WHERE aid = 474266;
UPDATE tellers SET tbalance = tbalance + 15549 WHERE tid = 19;
UPDATE branches SET bbalance = bbalance + 15549 WHERE bid = 5;
INSERT INTO history (tid, bid, aid, delta, mtime) VALUES (19, 5, 474266, 15549, CURRENT_TIMESTAMP);
END;
209

More info on PostgreSQL and DTrace

Look at Robert Lor's Blog
http://blogs.sun.com/robertlor/entry/user_level_dtrace_probes_in

For more examples see
http://pgfoundry.org/projects/dtrace/
210
DTrace & MySQL
Oh My My see what SQL I'm running
DTrace and MySQL



Some efforts in place to make embedded DTrace probes into
MySQL.
Until then here is a simple script to capture all the SQL statements
executed in a live running MySQL process
No extra recompiles needed.
212
DTrace and MySQL





Some efforts in place to make embedded DTrace probes into
MySQL.
Until then here is a simple script to capture all the SQL statements
executed in a live running MySQL process
No extra recompiles needed.
First find the demangled name for the function dispatch_command
()
dem `nm mysqld | cut -f 8 -d "|" ` | grep dispatch_command
__1cQdispatch_command6FnTenum_server_command_pnDTHD_pcI_b_ == bool dispatch_command
(enum_server_command,THD*,char*,unsigned)
213
DTrace and MySQL

Now for the D script
#!/usr/sbin/dtrace -qws
/*
* __1cQdispatch_command6FnTenum_server_command_pnDTHD_pcI_b_ == bool dispatch_c
ommand(enum_server_command,THD*,char*,unsigned)
*
*/
BEGIN
{
freopen("sql_commands_out");
}
pid$target::__1cQdispatch_command6FnTenum_server_command_pnDTHD_pcI_b_:entry
{
printf("%d:: %s\n",tid,copyinstr(arg2));
}
214
DTrace and MySQL

Output looks like
6590:: BEGIN
6590:: SELECT c from sbtest where id=503525
6590:: SELECT SUM(K) from sbtest where id between 497386 and 497485
6590:: SELECT c from sbtest where id between 537144 and 537243 order by c
6590:: SELECT c from sbtest where id between 499439 and 499538
6590:: SELECT DISTINCT c from sbtest where id between 502890 and 502990 order by c
6590:: UPDATE sbtest set k=k+1 where id=500085
6590:: UPDATE sbtest set c='183967968-592575299-975263686-423718108-495026220-90
1629681-234741050-54888582-190117389-133959759' where id=495351
215
DTrace & Oracle
A SQL of Oracle.
DTrace and Oracle



We can use the following simple D-script to show all SQL
statements executed by Oracle.
This is a internally developed script.
To run his script
# dtrace -32 -q -C -I. -s oracle32_queriesd pid-of-oracle-client

Disclaimers

This is assuming you are using 32 bit version of Oracle

The header file was constructed using a google for structure

This has been tested in Oracle 9i we can develop similar
scripts for later versions.
217
DTrace and Oracle
#include "oracle.h"
pid$1::sqlcxt:entry {
this->s = (struct sqlexd *) copyin(arg2,sizeof(struct sqlexd));
self->query = copyinstr((uintptr_t)this->s->stmt);
printf("%s: %s\n",execname, self->query);
}
218
DTrace and Oracle
static struct sqlexd {
unsigned int sqlvsn;
unsigned int arrsiz;
unsigned int iters;
unsigned int offset;
unsigned short selerr;
unsigned short sqlety;
unsigned int occurs;
short *cud;
unsigned char *sqlest;
char *stmt;
void *sqladtp;
void *sqltdsp;
void **sqphsv;
unsigned int *sqphsl;
int *sqphss;
void **sqpind;
int *sqpins;
unsigned int *sqparm;
unsigned int **sqparc;
unsigned short *sqpadto;
unsigned short *sqptdso;
void *sqhstv[4];
unsigned int sqhstl[4];
219
DTrace and Oracle



Also see script oracle32.d for an example of capturing time taken
for each query
You can use other standard pid provider instrumentation to look at
the oracle process
No need to recompile or restart oracle to use these scripts
220
Make your own
DTrace provider
DIY guide to providers
Defining Provider & Probes



Creating your own probe is very very simple.
Step 1: Figure out what probes you want to add to your app. This is
probably the hardest part. Remember

probes will be used by users who do not understand your
implementation details

probes can be listed so you need to think of how long you want to
support these probes.

probe and provider names should be intuitive

location of probe should be intuitive
You are more than 80% there.
222

Step 2: Provider & probe definition

create a .d file with the following entries
provider foobar {
probe foo(int,int);
probe bar();
};
You just created a provider foobar and two probes foo & bar. That's it.
(almost!)
The arguments are the types of the two arguments your probe exposes.
223

Step 3: Insert probes into your code.

in your source code at the location you want the probes to fire.
foo {
...
if(inp<10){
val1 = inp^3;
}else
val1 = inp^2;
}
...
}
Add the probe macro
#include <sys/sdt.h>
foo {
...
if(inp<10){
DTRACE_PROBE2(foobar, foo, inp, 3);
val1=inp^3;
}else
...
}
224

Step 4: Build your code.

Compile
cc -c probeale.c
dtrace -G -32 -s foobar.d probeable.o
cc -o probeable foobar.o probeable.o

The dtrace command compiles the .d file. It takes input from the
probeable.o(place in your code where you have added the code)

-G option generates a .o file

-32 / -64 for 32 and 64 bit apps.

The last line compiles all the .o's into your app.

Ok you are done! Really!
225
Using the probes


Now that you have created your own probes you can use them like
any other probe.
Some things to remember

Access your probe using <provider_name><pid> format.

So if your process id is 3346 then provider name is foobar3346
226
Details on DTrace
providers
More examples for providers.
Provider details
Here is some detailed info on providers
228
dtrace Provider
The dtrace provider provides three probes (BEGIN, END, ERROR)
BEGIN
BEGIN is the first probe to fire.
All BEGIN clauses will fire before any other probe fires.
Typically used to initialize.
END
Will fire after all other probes are completed
Can be used to output results
ERROR
Will fire under an error condition
For error handling

229
dtrace provider. Example
BEGIN
{
i = 0;
exit(0);
}
dtrace.d
ERROR
{
printf("Error has occurred!");
}
END
{
}
printf("Exiting and dereferencing a null pointer\n");
*(char *)i;
230
profile Provider




Profile providers has probes that will fire at regular intervals.
These probes are not associated with any kernel or user code
execution
profile provider has two probes. profile probe and tick probe.
format for profile probe: profile-n

The probe will fire n times a second on every CPU.

An optional ns or nsec (nano sec), us or usec (microsec), msec or
ms (milli sec), sec or s (seconds), min or m (minutes), hour or h
(hours), day or d (days) can be added to change the meaning of
'n'.
231
profile probe - examples

The following example prints out frequency at which proc execute
on a processor.
profile-100
{
@procs[pid, execname] = count();
}

prof.d
This one tracks how the priority of process changes over
time.
profile-1001
/pid == $1/
{
@proc[execname]=lquantize(curlwpsinfo->pr_pri,0,100,10);
}
prio.d
try this with a shell that is running...
$ while true ; do i=0; done
232
tick-n probe



Very similar to profile-n probe
Only difference is that the probe only fires on one CPU.
The meaning of “n” is similar to the profile-n probe.
233
lockstat Provider


lockstat has two kinds of probes. contention-event probes and holdevent probes.

contention-event – Used to track contention events. As these are
rare it does not impose too much of an overhead and so can be
safely enabled

hold-event – These are to track acquiring and releasing locks.
Enabling these probes can incur an overhead as these events are
more common.
lockstat allows you to probe adaptive, spin, thread and reader and
writer locks.
234
lockstat probes
235
lockstat - Example
Here is an example. It counts all the lock events of the given executable.
lockstat:::
/execname==$$1/
{
@locks[probename]=count();
}
lockstat.d
236
plockstat provider

One final provider that may be of interest is plockstat

plockstat is the user land equivalent of lockstat in kernel.
Three types of lock events can be traced.
Contention events – probes for user level lock contention
Hold events – probes for lock acquiring, releasing etc.
Error events – error coditions.

There are two families of probes
Mutex Probes
Reader Writer lock Probes

237
plockstat Providers
238
fbt Provider





The fbt – Function Boundary Tracing provider has probe into most
functions in the kernel.
Using fbt probe you can track entry and return from almost every
function in the kernel.
There are over 20,000 fbt probe in even the smallest Solaris systems
You'd need Solaris internal knowledge to be able to use this
effectively
Once opensolaris.org has entire Solaris code you will be able to use
these probes more effectively.

Very useful if you develop your own kernel module.

We will see a few examples.
239
fbt probe example.
The following example prints all the kernel functions called by ioctl
syscall from a “bash” shell.
#!/usr/sbin/dtrace -Fs
/*-F provides nice indented printing */
syscall::ioctl:entry
/execname == "bash"/
{
self->traceme = 1;
printf("fd: %d", arg0);
}
fbt1.d
fbt:::
/self->traceme/
{}
syscall::ioctl:return
/self->traceme/
{
self->traceme = 0;
exit(0);
}
240
proc Provider
The proc Provider has probes for process/lwp lifecycle
create – fires when a proc is created using fork and its variants
exec – fires when exec and its variants are called
exec-failure & exec-success – when exec fails or succeeds
lwp-create, lwp-start, lwp-exit – lwp life cycle probes
signal-send, signal-handle, signal-clear – probes for various signal
states
start – fires when a process starts before the first instruction is executed.

241
Examples
The following script prints all the processes that are created. It
also prints who created these process as well.
proc:::exec
{
self->parent = execname;
}
proc:::exec-success
/self->parent != NULL/
{
@[self->parent, execname] = count();
self->parent = NULL;
}
proc1.d
proc:::exec-failure
/self->parent != NULL/
{
self->parent = NULL;
}
END
{
printf("%-20s %-20s %s\n", "WHO", "WHAT", "COUNT");
printa("%-20s %-20s %@d\n", @);
}
242
More Examples
The following script prints all the signals that are sent in the
system. It also prints who sent the signal to whom.
proc:::signal-send
{
@[execname, stringof(args[1]->pr_fname),args[2]] = count();
}
proc2.d
END
{
printf("%20s %20s %12s %s\n", "SENDER", "RECIPIENT", "SIG", "COUNT");
printa("%20s %20s %12d %@d\n", @);
}
-bash-3.00$ ./proc2.d
^C
SENDER
sched
sched
sched
sched
bash
bash
RECIPIENT
dtrace
ls
bash
bash
bash
bash
SIG COUNT
21
21
18 4
25
25
20 12
243
sched Provider

The sched provider allows users to gain insight into how a process is
scheduled. It helps answer questions like why and when did the
thread of interest change priority.
The following are a few probes that are part of the sched provider.
change-pri – When priority changes
dqueue/enqueue – when proc taken off or put on the run queue
off-cpu / on-cpu – when thread taken off or put on a cpu.
preempt – when thread preempted
sleep / wakeup – when thread sleep on a synchronization object and
when it wakes up.

244
sched examples.
This script prints the distribution of the time threads spends on a
cpu.
sched:::on-cpu
{
self->ts = timestamp;
}
sched.d
sched:::off-cpu
/self->ts/
{
@[cpu] =quantize(timestamp - self->ts);
}
245
DTrace data structure
A peek into more that can be done with
DTrace
Arrays
name[key] = expression;

name – name of array

key – list of scalar expression values (tuples)

expression – evaluates to the type of array
{
ts[probefunc,pid,tid]=timestamp; /* save time stamp at entry */
}
array.d
{
timespent = timestamp - ts[probefunc,pid,tid];
printf("%s threadID %d spent %d nsecs in %s\n", execname, tid, timespent, probefunc);
/* print time-spent at return */
ts[probefunc,pid,tid]=0;
timespent = 0;
}
247
struct construct
struct type{
element1;
element2;
}
Example
struct info{
string f_name;
int count;
int timespent;
} /* definition of struct info */
struct info my_callinfo; /* Declaring my_callinfo as variable of type info */
my_callinfo.f_name; /* access to member of struct */
248
Postmortem Tracing
Cold cases!
Postmortem tracing


A nifty feature of DTrace is to be able to dig dtrace related info from a
system crash dump.
Feature could be very useful to support engineers
Here is how it works.

Load core dump into mdb
> ::dtrace_state – prints out details about all dtrace consumers when the
dump was generated.

Take the address for dtrace consumer and
> <addr>::dtrace – prints all the info from dtrace buffer.

250
Postmortem tracing



You can create a ring buffer of data using dtrace

Use the -b option for data size & -bufpolicy=ring for ring buffer
policy.
You can leave this running and if system crashes you can analyze
the buffer from the crash dump.
Options

<addr>::dtrace -c 1
 Print only info from cpu 1.
251
Speculative tracing
DTrace meets Las Vegas
Speculation

We will now see how to catch a bad code path using speculation!
Here is why we need speculation.

Some time we only see error message after the error has occured.
For example: We see a function return an error but the problem was
caused by something that the function did earlier.
We see the error and want to go back and find out what the function did
wrong. But alas the function has already happened

One solution could be to save details every time the function
executes but this is wasting trace buffer with a lot of useless data
when were are only concerned about the one time the function
failed.

A better solution - speculation

253
Speculation - example
pid$target::fopen64:entry
{
self->spec = speculation();
speculate(self->spec);
printf("Path is %s\n", copyinstr(arg0));
}
pid$target:::entry
/self->spec/
{
}
pid$target:::return
/self->spec/
{
}
spec.d
pid$target::fopen64:return
/self->spec && arg1 != 0/
{
discard(self->spec);
self->spec = 0;
}
pid$target::fopen64:return
/self->spec && arg1 == 0/
{
commit(self->spec);
self->spec = 0;
}
254
DTrace
configuration
Fine tune the tool!
Granting privilege to run dtrace

A system admin can grant any user privileges to run dtrace using
the Solaris Least Privilege facility privileges(5).
DTrace provides for three types of privileges.
dtrace_proc - provides access to process level tracing no kernel
level tracing allowed. (pid provider is about all they can run)
dtrace_user – provides access to process level and kernel level
probes but only for process to which the user has access. (ie) they
can use syscall provider but only for syscalls made by process
that they have access.
dtrace_kernel – provides all access except process access to user
level procs that they do not have access.


Enable these priv by editing /etc/user_attr.

format user-name::::defaultpriv=basic,privileges
256
Increasing max probes


You can see how easy it is to create a lot of probes on the fly using
the pid provider. Just one note before we move on.
By default the maximum probes that Solaris allows is 250,000. You
can increase this by editing /kernel/drv/fasttrap.conf

increase fasttrap-max-probes variable

run update_drv fasttrap or reboot system
257
Consumers



A DTrace consumer is a process that interacts with
DTrace
There is no limit on concurrent consumers
dtrace(1M) is a DTrace consumer that acts as a
generic front-end to the DTrace facility
Examples:
sysinfo, lockstat and plockstat.

258
DTrace more
resources
Yes there is more to DTrace than can be
captured in 200 slides
DTrace Resources


Here are a few of the many DTrace resources available for you
“Solaris Dynamic Tracing Guide” is an excellent resource. Many of the
examples in this presentation are from the Guide. http://
docs.sun.com/db/doc/817-6223
OpenSolaris DTrace community is one of the best places to get an
answer on anything DTrace.
http://opensolaris.org/os/community/dtrace/
Forum: http://www.opensolaris.org/jive/forum.jspa?forumID=7

The Big admin DTrace web page has a lot of good info
http://www.sun.com/bigadmin/content/dtrace/

260
More DTrace Resources
Read the Blogs from Bryan Cantrill, Adam Leventhal, Mike Shapiro
http://blogs.sun.com/roller/page/bmc
http://blogs.sun.com/roller/page/ahl
http://blogs.sun.com/mws
They often speak about DTrace related issues.

Email me if there is anything I can help you with.
[email protected]
Of course you can google DTrace.
http://www.google.com/search?q=dtrace

261
Angelo Rajadurai
[email protected]

Lab presentation - Observing and Optimizing your Application with

Transcription

Similar documents

DTrace - Black Hat

elotest b300

Downtown San Angelo - Standard

Selecting Quality Child Care: A Parents Handbook

the Inside Edition - Eastern Illinois University

Probe Year in Review

Technical data sheet Electro ejaculator

CRYOGEN-FREE Models

using standard probes with the tektronix 222ps oscilloscope