Lots of FreeBSD users are coming to ZFS since the release of FreeBSD 10, as it's so easy to install the system on top of that powerful filesystem. ZFS default behavior and settings are perfect for a wide range of workloads and uses, but it's not exactly what you need for databases hosting.
I'm running a FreeBSD 9.x server hosting http, php, mysql, mail, and many other things, so a databases-only optimization would be counterproductive. After a good dive into experts do's & don't's here are few steps I've taken to tune my server.
Datasets
If like me you've started hosting MySQL databases a long time ago (say ~15 years ago) you might have some DB using the old MyISAM engine, and some newer ones using InnoDB. Guess what. They use different block sizes, and they use different caching mechanisms.
On top of that, the block size for InnoDB is not the same when you deal with data files, and when you deal with log files.
In order to get the best block size and cache tuning possible, I've created 3 datasets: one for InnoDB data files, one for innodb logs, and one for everything else, including MyISAM databases.
Block size
The block size is engine dependent. MyISAM uses a 8k block size, InnoDB uses a 16k block size for data and 128k for logs. It's easy to setup datasets for a particular block size at creation with -o
option, or after creation with zfs set
. You must set the proper block size before putting any data on the dataset, otherwise pre-existing data won't use the desired block size.
Caching
MyISAM engine relies on the underlying filesystem caching mechanism, so you must ensure ZFS will cache both data and metadata (that's the default behavior). On the other hand, InnoDB uses an internal cache, so it would be a waste on memory to cache the same data into ZFS and InnoDB. On a dedicated MySQL server, the proper tuning would require to limit ARC size, and to disable data caching in ZFS. On a general-purpose server, ARC size should not be tweaked, but on InnoDB dedicated datasets it's easy to disable ZFS cache for data by setting the property primarycache
to "metadata
".
On MySQL's side
You must of course tell mysqld
where to find data files and logs. I've set those values in my /var/db/mysql/my.cnf
file:
innodb_data_home_dir = /var/db/mysql-innodb innodb_log_group_home_dir = /var/db/mysql-innodb-logs
where /var/db/mysql-innodb
and /var/db/mysql-innodb-logs
are mount points for the dataset dedicated to InnoDB data files, and the dataset dedicated to InnoDB log files.
As my zpool is a mirror, I've added this setting too:
skip-innodb_doublewrite
Step by step
I've followed this course of action:
1st step: mailing to users, "MySQL will be unavailable for about 5 minutes, hang on."
2nd step: backup /var/db/mysql
and edit your my.cnf
3rd step: shutdown your mysql server
sudo service mysql-server stop
4th step: move /var/db/mysql
to /var/db/mysql-origin
5th step: create appropriate datasets:
zfs create -o recordsize=16k -o primarycache=metadata zmirror/var/db/mysql-innodb zfs create -o recordsize=128k -o primarycache=metadata zmirror/var/db/mysql-innodb-logs zfs create -o recordsize=8k zmirror/var/db/mysql
6th step: move data from /var/db/mysql-origin
to your new datasets
cd /var/db/ sudo mv mysql-origin/ib_logfile* mysql-innodb-logs/ sudo mv mysql-origin/ibdata1 mysql-innodb/ sudo mv mysql-origin/* mysql/
and set proper rights:
sudo chown mysql:mysql mysql-innodb-logs mysql-innodb mysql sudo chmod o= mysql mysql-innodb-logs mysql-innodb
7th step: restart your mysql server
sudo service mysql-server start & tail -f mysql/${HOSTNAME}.err
8th step: mailing to users, "MySQL's back online maintenance duration 5 min 14 sec."
Further reading & references
MySQL Innodb ZFS Best Practices
A look at MySQL on ZFS
Optimizing MySQL performance with ZFS
ZFS for Databases