Best Practices For Deploying MySQL on Solaris Presented by,

Best Practices For
Deploying MySQL on
Solaris
Presented by,
MySQL AB® &
O’Reilly Media, Inc.
Ritu Kamboj
Jenny Chen
Agenda
 MySQL -Solaris Integration
 MySQL High Availability Data Service
 Consolidate MySQL Deployment Using Solaris
Containers
 DTrace (Jenny Chen)
 MySQL-Solaris Integration
Optimized MySQL on Open Solaris
 MySQL 5.0.45 (32bit) integrated with Open
Solaris build 76
 SXDE 01/08
 MySQL 5.0.45 (64bit) integrated with Open
Solaris build 87
 SXCE
Layout of MySQL on Open Solaris
 MySQL 5.0.45 packages
 Default data directory
 /var/mysql/5.0/data
 Default configuration directory
 /etc/mysql/5.0
 Installation directory
 /usr/mysql/5.0/
 Latest version accessible from /usr/mysql/bin
 Symbolic link from all directories involved in 5.0.45
release to corresponding /usr/mysql
Optimization Of MySQL On Open Solaris
 Compiled with optimal compiler options using
Sun Studio compilers
 Added SMF support for MySQL
 Easily accommodates varying configurations
 Initializes MySQL database thereby enhancing user
experience
 Mysql user can manage MySQL database via SMF
 Support for MySQL cluster engine (ndb)
Compiler Optimization
 Enable in-lining
 Change header file univ.i to add Sun Studio
 If (!defined(SUNPRO_C)
 #define UNIV_MUST_NOT_INLINE
 Enable pre-fetching
 -xprefetch=auto and -xprefetch_level=3
 Set optimization level
 -xO4
Compiler Optimization (Contd)
 Feedback optimization (not yet implemented)
 Workload is key
 -xprofile -xipo
 Interprocess optimization
 About 10% improvement
 Link with libmtmalloc
 Library for threaded application
 About 8% improvement
Service Management Facility (SMF)
 Makes Solaris Services Self-Healing
 Services automatically restart in dependency
order
 Misbehaving and mis-configured services are
easier to debug
 Log files for each service
 Administrators can securely delegate tasks to
non-root users
MySQL-Service Management Facility
 Dynamically configured properties
 mysql/bin
 Installation , default : /usr/mysql/5.0/bin
 mysql/data
 Data Directory , Default: /var/mysql/5.0/data
 mysql/enable_64bit
 Flag to select 32bit or 64bit , Default : false
 First time installations
 Creates system tables (mysql_install_db.sh )
Starting MySQL on Open Solaris
 Older Open Solaris Build (Build 76- Build-87)
 32Bit MySQL integrated
 Create mysql user
 Enable MySQL SMF service
 Latest Open Solaris Build (Build 87 onwards)
 32Bit and 64bit MySQL integrated
 Default mode : 32bit service
 For starting 64bit
 Set enable_64bit ==
true
Optimized MySQL on Solaris 10
 CoolStack
 SAMP stack + more
 Optimized MySQL download
 Standalone package
 SAMP stack component
 Available at Sun Download center:
 http://cooltools.sunsource.net/coolstack
 Version MySQL 5.0.45
 Similar optimizations as in Open Solaris
 MySQL-High Availability Data Service
Solaris Cluster Overview
 Provides general purpose HA platform
Availability is our customers'
most critical requirement
Sun Cluster VOC Survey
Fifty percent of enterprises that lack a
recovery plan go out of business
within
one year of a significant disaster
Gartner Group
Solaris Cluster : Hardware Components
 Servers with local storage
 Can have up to 16 nodes
 Shared storage
 Tolerates single-node failures
 Centralizes configuration files
 Cluster interconnect
At least two redundant networks
 Public network interfaces
 Spreads outbound packets
Solaris Cluster Algorithms
 Cluster membership monitor
 Ensures data integrity
 Determines cluster membership
 Cluster configuration repository
 Global repository
 Ensures consistent view
 Disk Fencing
 Fences off non-cluster nodes
 Prevents Partition
 Quorum
 Uses a majority voting schema
MySQL High Availability Data Service
Node 4
Node 3
Node 1
D
B
Node 2
 HA-MySQL is a failover data service
MySQL High Availability Data Service
 Supported configurations
 Standalone MySQL server
 MySQL replication server
 Single/Multiple MySQL instances in master configuration
 Single/Multiple MySQL instances in slave configuration
 Solaris containers support
Global zone
 Non-global failover zone
 Non-global zone
HA-MySQL Service Deployment
MySQL
Stor
Node1
MySQL
Host
Node2
Stor
Node1:Zone1
Host
Node2:Zone2
MySQL-Solaris Cluster Benefits
 Enhanced end-to-end infrastructure availability
 Continuous MySQL Availability
 Automatic failover if master node fails
 Low cost solution
 Software is free and open sourced
 Efficient Resource Utilization
 Multiple applications can be consolidated
 Ease of operations
 SC enables clustered systems to be managed as if
they were on a single system
Additional information
 Step by step deployment guide
 http://docs.sun.com/app/docs/doc/819-3059
 Failover study of HA-MySQL
 http://blogs.sun.com/krishs/date/200804
 Solaris Cluster
 http://www.sun.com/software/solaris/cluster/index.xml
http://opensolaris.org/os/community/ha-clusters/
Consolidate MySQL installations using Solaris
Containers
Solaris Containers
 Containers : Zones + Resource Management
 Zones: isolated virtual application environments
 Resource management – resource control (CPU,
Memory)
 Achieving Consolidation Goals
 Reduce Hardware
 Combine low utilization systems
 Isolate applications from faults
 Maintain Service Levels
 Fine tune response times
MySQL Consolidation Study
 System Configuration
Sun Fire X4100 (4 CPU, 8 GB Memory)
 Local container configuration ( 1 CPU , 1 GB
Memory
 Sysbench Read-only
Sysbench read-write
Sysbench read-only
Sysbench read-write
675
1900
650
625
600
1700
1600
No Containers
With Containers
1500
Throughput
Throughput
1800
575
550
No Containers
With Containers
525
500
475
1400
450
1300
425
32
64
128
Number of threads
256
32
64
128
Number of threads
256
 Dtrace (Jenny Chen)
DTrace
 Use DTrace with MySQL to drill down
Monitoring MySQL Performance
 Examples & Solutions
 Easy Steps To add DTrace probes into
MySQL core server and storage Engines
 Easy Steps to display MySQL DTrace
probes into Chime visualization Tool for
DTrace
Why DTrace
 Solaris 10 Dynamic Tracing Facility to provide
comprehensive view of operating system and
application behaviour
> DTrace to examine particular system areas: disk I/O, CPU,
Memory
> Process Tracing and Debugging
USDT(User-level statically defined tracing) place custom probes in
application code
Add USDT into MySQL source to monitor MySQL and gather the
useful data missing by the current MySQL monitor tools:
DTrace: Monitor I/Os

Exam I/O wait time by filename and mysqld(Available at DTraceToolkit)
#!/usr/sbin/dtrace –s
#pragma D option quiet
io:::wait-start
/ execname == “mysqld” /
{ self->start = timestamp; }
io:::wait-done
/ execname == “mysqld” && self->start /
{
this->elapsed = timestamp - self->start;
@files[pid,args[1]->dev_pathname, args[2]->fi_pathname] = sum(this->elapsed);
self->start = 0;
}
profile:::tick-5s
{ printf(“-------------------------------------------------\n”);
printf(“%6s %8s %20s %50s\n”, “PID”, “TIME”, “DEVICE”, “FILE”);
printa(“%6d %@8d %20s %8s \n”, @files);
printf (“------------------------------------------------\n”); }
#./mysqliowait.d
--------------------------------------------------PID
TIME DEVICE
113
234
/dev/dsk/c0t2d0s0
FILE
/usr/local/mysql/data/ibdata1
DTrace: Monitor CPU

classic performance problem
#!/usr/sbin/dtrace -s
#pragma D option quiet
syscall:::entry
{
self->ts=vtimestamp;
}
syscall:::return
/self->ts/
{
@a[execname, probefunc] = count();
@b[execname, probefunc] = sum (vtimestamp - self->ts);
self->ts=0;
}
END
{
printf("%-16s %-16s %-8s\n","EXEC","SYSCALL","COUNT");
printa("%-16s %-16s %-@8d\n",@a);
printf("%-16s %-16s %-8s\n","EXEC","SYSCALL","TIME");
printa("%-16s %-16s %-@8d\n",@b);
}
# ./syscall.d
EXEC
SYSCALL
COUNT
devfsadm
lwp_park
1
dtrace
fstat
1
...
mysqld
read
106542
mysqld
gtime
109613
mysqld
pread
1181669
--------------------------------------------------EXEC
SYSCALL
TIME
dtrace
lwp_sigmask
218
pkill
getpid
302
...
mysqld
read
259284183
mysqld
write
267556239
mysqld
pread
4650457224

Agrregated user stack backtrace to understand of the nature of pread() in
MySQL source code – useful for mysql developers
# dtrace -n 'syscall::pread:entry / execname == "mysqld" / { @[ustack()]=count() }
dtrace: description 'syscall::pread:entry ' matched 1 probe
...
libc.so.1`_pread+0xa
mysqld`my_pread+0x54
mysqld`_mi_read_static_record+0x67
mysqld`mi_rnext+0x1fe
...
mysqld`handle_one_connection+0x855
libc.so.1`_thr_setup+0x67
1564811
....
Replace high cost pread with
“--myisam_use_mmap=1”- useful for mysql DBA

A 94% performance
improvement !
Dtrace: Memory Analysis

Check which process causes anonymous page in
# dtrace -n anonpgin '{@[execname] = count()}'
dtrace: description anonpgin matched 1 probe
sshd
2
vmstat 23
mysqld 673
Use Dtrace to measure waiting for paging in
# ./ whospaging.d – available at Solaris Internals(http://www.solarisinternals.com/si/dtrace/)
Who's on cpu (milliseconds):
sshd
1
vmstat
3
mysqld 120
sched
43210
Who's waiting for pagin (milliseconds):
mysqld
239082
Dtrace Probes In MySQL

Provide deep view of internal MySQL core server and storage engines'
operation & behaviour
>Database information
>Query execution latency
>Index & table scan cost
>Wait events inside MyISAM & Innodb
>Deadlock information
>Query cache hit/miss
>And many more...

Speed resolution of performance bottlenecks with in database design
and MySQL server configurations

Negligible performance overhead

Easy steps to create & insert your own Dtrace probes into MySQL

MySQL DTrace GUI Monitor Tool - Chime
Query execution Time
 Enable Slow query log with “–log-slow-queries” requires re-start
MySQL server
 SQL statements with query execution time longer than
“long_query_time” second in the log file
 SQL statements generating most loads on the application may
not in slow query log
 Replication query statements are not available in slow query log
 Time spending by the query optimizer to generate query plan is
not available in slow query log
 Using DTrace can get mising query execution information online
Insert DTrace Probes Into MySQL
 Step 1: Figure out what probes are needed to insert into the
source code
 Step 2: Define MySQL Provider and probes
# cat mysql_dtrace.d
provider mysql
{
probe query__execute__start(void *, char *, char *, const char *, char *);
probe query__execute__finish(void *, char *, char *, const char *, char *,int);
}
>Two Probes defined in the mysql provider
>Note to use two underscore(__) translated to hypen automatically
 Step 3: Define a header file “mysql_dtrace.h” with definitions for
probes
dtrace -h -s mysql_dtrace.d
#ifndef _MYSQL_H
#define _MYSQL_H
#define DTRACE_QUERY_EXECUTE_START(arg0, arg1, arg2, arg3, arg4) \
__dtrace_mysql___query__execute__start(arg0, arg1, arg2, arg3, arg4)
#define DTRACE_QUERY_EXECUTE_START_ENABLED() \
__dtraceenabled_mysql___query__execute__start()
extern void __dtrace_mysql___query__execute__start(void *, char *, char *, char
*, char *);
extern int __dtraceenabled_mysql___query__execute__start(void);
#endif
 Step 4: Insert the probes into source code
#include <mysql_dtrace.h>
...
bool mysql_execute_command(THD *thd)
{
DTRACE_QUERY_EXECUTE_START((void *)thd, thd->db,
thd->security_ctx->user, (char *)thd->security_ctx->host_or_ip,thd->query);
...
DTRACE_QUERY_EXECUTE_FINISH((void*)thd, thd->db,
thd->security_ctx->user, (char *)thd->security_ctx->host_or_ip,
thd->query, res ==0 ? 0: -1);
go to end;
...
}
 Step 5: Build MySQL with DTrace
 In the Makefile.in, compile 64-bit MySQL with Dtrace
mysqld_OBJECTS = $(am_mysqld_OBJECTS) mysql_dtrace.$(OBJEXT)
mysql_dtrace.o:$(top_srcdir)/include/mysql_dtrace.d $(am_mysqld_OBJECTS)
dtrace -G 64 -s $(top_srcdir)/include/mysql_dtrace.d $(am_mysqld_OBJECTS)

Inserting DTrace probes comleted, DTrace probes are ready to use!
 Step 6: Use inserted DTrace probes to measure query execution
time with other database information
#!/usr/sbin/dtrace -s
…
mysql*:::query-execute-start
{ self->start = timestamp;
}
mysql*:::query-execute-finish
/self->start/
{ this->query = copyinstr(arg4); }
mysql*:::query-execute-finish
/self->start/
{
this->elapsed = (timestamp - self->start) / 1000000;
this->who = strjoin(copyinstr(arg2), strjoin("@", copyinstr(arg3)));
printf(" %-16.16s %-18.18s %5d %3d %-32.32s\n", arg1 ? copyinstr(arg1)
: ".", this->who, this->elapsed, (int)arg5, this->query); self->start = 0;
}

# ./mysqld_qestat.d
DATABASE
sbtest
sbtest
sbtest

USER@HOST
root@localhost
root@localhost
root@localhost
ms RET QUERY
0
0
0 0
178 0
show tables
show databases
select * from sbtest
Use the same steps to insert Dtrace probes to
<source_tree>/sql/sql_select.cc at the start and end of choose_plan()
function to measure the time spent in query optimization

optimizer_prune_level=1 reduce query compilation time

Reduce optimizer_search_depth or optimizer_search_depth=0
Index & Table scan cost

Identify the places in MySQL source to handle scanning index, and
table
>Index-scan functions: index_next, index_next_same, index_prev,
index_first, index_last
>Table-scan functions: rnd_init, rnd_end, rnd_next, rnd_pos
>Insert DTrace Probes at the start and before return from the functions
can measure the time spending on scanning table or index.
mysql*:::innodb-index-next-start
{ @indexnext[args[0]] = count();
self->inext = timestamp; }
mysql*:::innodb-index-next-finish
/self->inext/
{ @indexnexttime[args[0]] = sum(timestamp - self->inext);
self->inext = 0; }

Expensive index-scan/table-scan report from Dtrace requires to
optimize schema accordingly
Buffer wait in Innodb

Innodb buffer wait is common in I/O-bound MySQL system while reading
page synchronous from disk
Insert Dtrace probes at: innobase/buf/buf0rea.c,
ulint buf_read_page( ulint space, ulint offset)
{ ...
DTRACE_INNODB_BUFFER_WAIT_START();
/* We do the i/o in the synchronous aio mode to save thread
switches: hence TRUE */
count2 = buf_read_page_low(&err, TRUE, BUF_READ_ANY_PAGE, space,
tablespace_version, offset);
DTRACE_INNODB_BUFFER_WAIT_FINISH();
...
}
 Increase innodb_buffer_pool size
 Tune SQL to access rows with fewer block reads(i.e. By adding indexes)
Query cache probes

Use DTrace to measure query cache hit and query cache miss to
determine how well the query cache is performing
> Insert DTrace Probes at: sql/sql_cache.cc function: send_result_to_client
> DTrace Test script to report query cache hit and miss counts by query string, and
totals
mysql*:::query-cache-hit,
mysql*:::query-cache-miss
{ this->query = copyinstr(arg4); }
mysql*:::query-cache-hit
{ @elapsed[this->query, "hit"] = count();
hits++; }
mysql*:::query-cache-miss
{ @elapsed[this->query, "miss"] = count();
misses++; }
# ./mysqld_qchit.d
Tracing... Hit Ctrl-C to end.
^C
QUERY
RESULT COUNT
select * from months
miss
1
select * from months where num > 3
hit
1
select * from months where num > 3 and num < 9
miss
1
show databases
miss
1
show tables
miss
1
select * from months
hit
9
Hits
: 10
Misses : 4
Hit Rate : 71%

Tuning “query_cache_size” variable according to the hit/miss rate
DTrace Performance Impact
 Inserting DTrace Probes into MySQL source code are useful for
MySQL DBA, MySQL & application developers. The
performance impact of adding DTrace probes is critical for
enterprise environment.
 Cost of inserting USDT probe can be basically negligible:
Each probe inserted into the source code can be enabled by adding the
code like:
if (PROVIDER_PROBE_ENABLED()
{
PROVIDER_PROBE(arg0,...);
}
DTrace Probes In MySQL 6.0
probe insert_row_start();
probe insert_row_end();
probe filesort_start();
probe filesort_end();
probe delete_start();
probe delete_end();
probe select_start()
probe select_end();
probe update_start();
probe update_end();
 compile with “--enable-dtrace” configure option to use
the Dtrace probes
Integrate with Chime Tool
 Chime is graphical tool for visualizing DTrace aggregations. It
provides alternative CLI-based tool output with more visually
apprealing and more useful to display data over time
 Available to download:
http://www.opensolaris.org/os/project/dtrace-chime. One-step
installation:
> Run pkgadd -d osol0chime-<arch>-1.4.pkg
 chime requires Solaris Nevada build 35 or later
> Run local: /opt/OSOL0chime/bin/chime
> Run remotely: /opt/OSOL0chime/bin/chime <hostname>
 chime provides wizard to automatically generate new display for
DTrace script
 Resources
> http://www.opensolaris.org/os/community/dtrace
- OpenSolaris Community: Dtrace
> http://developers.sun.com/solaris/articles/solaris_perftools.html
- Solaris Performance and Tools
> http://docs.sun.com/app/docs/doc/817-6223/6mlkidlms?a=view
- Statically Defined Tracing for User Applications chapter of DTrace
Manual
> http://www.brendangregg.com
- DTrace toolkits
>
http://www.solarisinternals.com/wiki/index.php/DTrace_Topics_Databases
- DTrace Topics Databases
 Acknowledgements
Brandan Gregg – Sun Microsystems Engineer in Advanced Products
Group