Dtrace on FreeBSD 9.2

FreeBSD logoAs of FreeBSD 9.2-RELEASE, Dtrace is enabled by default in the GENERIC kernel. We where already able to activate Dtrace, on earlier releases, but we would have to make a custom kernel. Not always what you want because it breaks binary updates.

Now that Dtrace is enabled (and updated) in FreeBSD, everyone can use it without the hassle of a custom kernel. You still have to load Dtrace's modules. Let's see how it works.

first you go root and try a Dtrace related command:

$ sudo -Es
# dtruss ls
dtrace: failed to initialize dtrace: DTrace device not available on system

Looks like Dtrace is not loaded:

# kldstat 
Id Refs Address            Size     Name
 1   23 0xffffffff80200000 15b93c0  kernel
 2    1 0xffffffff817ba000 23d078   zfs.ko
 3    2 0xffffffff819f8000 84e0     opensolaris.ko
 4    1 0xffffffff81a01000 4828     coretemp.ko
 5    1 0xffffffff81c12000 2bce     pflog.ko
 6    1 0xffffffff81c15000 3078d    pf.ko
 7    1 0xffffffff81c46000 57bcf    linux.ko
 8    1 0xffffffff81c9e000 3c3d     wlan_xauth.ko

lets find Dtrace related modules:

# ls /boot/kernel/dtrace*
/boot/kernel/dtrace.ko			/boot/kernel/dtrace_test.ko.symbols
/boot/kernel/dtrace.ko.symbols		/boot/kernel/dtraceall.ko
/boot/kernel/dtrace_test.ko		/boot/kernel/dtraceall.ko.symbols

/boot/kernel/dtraceall.ko looks like a winner.

# kldload dtraceall
# kldstat 
Id Refs Address            Size     Name
 1   59 0xffffffff80200000 15b93c0  kernel
 2    1 0xffffffff817ba000 23d078   zfs.ko
 3   16 0xffffffff819f8000 84e0     opensolaris.ko
 4    1 0xffffffff81a01000 4828     coretemp.ko
 5    1 0xffffffff81c12000 2bce     pflog.ko
 6    1 0xffffffff81c15000 3078d    pf.ko
 7    1 0xffffffff81c46000 57bcf    linux.ko
 8    1 0xffffffff81c9e000 3c3d     wlan_xauth.ko
 9    1 0xffffffff81ca2000 ba2      dtraceall.ko
10    1 0xffffffff81ca3000 4ed3     profile.ko
11    3 0xffffffff81ca8000 402c     cyclic.ko
12   12 0xffffffff81cad000 23dbaa   dtrace.ko
13    1 0xffffffff81eeb000 fb3d     systrace_freebsd32.ko
14    1 0xffffffff81efb000 109bd    systrace.ko
15    1 0xffffffff81f0c000 45bb     sdt.ko
16    1 0xffffffff81f11000 4926     lockstat.ko
17    1 0xffffffff81f16000 bf0b     fasttrap.ko
18    1 0xffffffff81f22000 6673     fbt.ko
19    1 0xffffffff81f29000 4eeb     dtnfsclient.ko
20    1 0xffffffff81f2e000 1dd92    nfsclient.ko
21    1 0xffffffff81f4c000 47c0     nfs_common.ko
22    1 0xffffffff81f51000 55f2     dtnfscl.ko
23    1 0xffffffff81f57000 45cd     dtmalloc.ko
24    1 0xffffffff81f5c000 44fc     dtio.ko


# dtruss ls
SYSCALL(args) 		 = return
mmap(0x0, 0x8000, 0x3)		 = 6418432 0
issetugid(0x0, 0x0, 0x0)		 = 0 0
lstat("/etc\0", 0x7FFFFFFFC3C0, 0x0)		 = 0 0
lstat("/etc/libmap.conf\0", 0x7FFFFFFFC3C0, 0x0)		 = 0 0
open("/etc/libmap.conf\0", 0x0, 0x81FA20)		 = 3 0

Happy debugging/tracing/auditing.

Munin plugins for CRM114

I'm using CRM114-based SpamAssassin plugin for spam filtering at work, and at home. Client-side, I'm able to check CRM114 contribution by a simple look at headers of an email message. But that won't tell you how CRM114 is behaving on the server side. The main concern on the server, is to check the two "databases" spam.css and nonspam.css.
I've chosen to monitor both average packing density and documents learned metrics for both files. The first one goes from 0 to 1. When its value reaches 0.9 and beyond, you must make sure your antispam filtering does not become sluggish.
The second one represents the number of emails from which CRM114 has been trained. Basically, CRM114 is only trained from its mistakes : if an email is flagged as spam by mistake, you can train CRM114 to learn it as nonspam, but if an email is properly flagged as spam, you can't train CRM114 to learn it as spam.

I've designed two Munin plugins, they run on FreeBSD but portage to another UNIX is just a matter of path.

Monitor "average packing density" (crm114_packingdensity): shows how much your CRM114 bases are encumbered

# Parameters:
# 	config   (required)
# 	autoconf (optional - used by munin-config)
# Magick markers (optional - used by munin-config and som installation
# scripts):
#%# family=auto
#%# capabilities=autoconf

export PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# config
if [ "$1" = "config" ]; then
    echo 'graph_title CRM114 css file stat average packing density'
    echo 'graph_vlabel Packing density'
    echo 'graph_category ANTISPAM'
    echo 'graph_args --upper-limit 1 --lower-limit 0'
    echo 'graph_info This graph shows the average packing density for CRM114 css files'
    echo 'spam.label spam'
    echo 'nonspam.label nonspam'
    exit 0

cssutil -r -b /var/amavis/.crm114/spam.css | awk '/Average packing density/ {print "spam.value "$5}'
cssutil -r -b /var/amavis/.crm114/nonspam.css | awk '/Average packing density/ {print "nonspam.value "$5}'

Monitor "documents learned" (crm114_documentslearned): shows how many emails CRM114 has learned from

# Parameters:
# 	config   (required)
# 	autoconf (optional - used by munin-config)
# Magick markers (optional - used by munin-config and som installation
# scripts):
#%# family=auto
#%# capabilities=autoconf

export PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# config
if [ "$1" = "config" ]; then
    echo 'graph_title CRM114 css file stat documents learned'
    echo 'graph_vlabel Documents learned'
    echo 'graph_category ANTISPAM'
    echo 'graph_args --lower-limit 0'
    echo 'graph_info This graph shows the documents learned count for CRM114 css files'
    echo 'spam.label spam'
    echo 'nonspam.label nonspam'
    exit 0

cssutil -r -b /var/amavis/.crm114/spam.css | awk '/Documents learned/ {print "spam.value "$4}'
cssutil -r -b /var/amavis/.crm114/nonspam.css | awk '/Documents learned/ {print "nonspam.value "$4}'

Both plugins need read access to spam.css and nonspam.css, depending on your setup it will require special privileges. You might want to add in /usr/local/etc/munin/plugin-conf.d/plugins.conf:

user root

Running source dedicated server on FreeBSD 9.x

steam logo © steamI've covered this subject in french back in 2010, but things have evolved, and installing scrds on FreeBSD is not as straightforward as it used to be. Prerequisites are the same, you must first install the linux compatibility layer:

# As root, load the module & make sure it will be loaded after reboot:
kldload /boot/kernel/linux.ko
echo 'linux_enable="YES"' >> /etc/rc.conf

Then, and only after loading the linux module, install linux_base:

portinstall -PP linux_base-f10
(or portinstall linux_base-f10 if the command above fails)

Add linproc to /etc/fstab,

linproc         /compat/linux/proc      linprocfs       rw 0 0

and mount it:

mount -a

Install linux-steam:

portinstall linux-steam

Add a steamuser user (or whatever name you want). The user must be unprivileged, and must not be able to log into the machine. Set its home to /usr/local/steam and its shell to /usr/sbin/nologin.
Then, update the client linux-steam:

chown -R steamuser /usr/local/steam
cd /usr/local/steam
sudo -u steamuser ./steam
sudo -u steamuser ./steam (yes, do it twice, to make sure it's up to date)

Back in past, you would have used the steam client to install and update games, but people at steam thought it was way too easy. So they made it more complicated. Now you have to install a dedicated tool in order to install games and keep them updated: steamcmd.
Point your browser to the SteamCMD wiki page at Valvesoftware and read it. Then, download the linux version and extract in a dedicated directory (/usr/games for example).

cd /usr/games
fetch http://media.steampowered.com/client/steamcmd_linux.tar.gz
tar -xzf steamcmd_linux.tar.gz

By default, the steamcmd command will maintain your game library into /usr/local/steam/Steam/SteamApps, you might want to create a soft link in order to put your game library somewhere else (the force_install_dir option of steamcmd would not work on my server). At least, change owners for /usr/games/SteamCMD directory, to make sure steamuser can update its content:

mkdir /usr/local/steam/Steam
mkdir /usr/games/SteamApps
ln -s /usr/games/SteamApps /usr/local/steam/Steam/SteamApps
chown steamuser /usr/games/SteamApps
chown -R steamuser /usr/games/SteamCMD

Then you might have to change shebangs in SteamCMD/steam.sh and SteamCMD/steamcmd.sh to use your own bash (probably /usr/local/bin/bash instead of /bin/bash). After what you can launch steamcmd.sh:

cd /usr/games/SteamCMD
sudo -u steamuser ./steamcmd.sh

The program should auto-update, and present you with a Steam> prompt.
Then, you must login. For L4D2, and most games, you can login anonymously:

login anonymous

Choose your game from the list on the wiki, and use its ID to install/update:

app_update 222860 validate

It's possible to automate SteamCMD, for daily update of your games. For example you can create a shell script like this one:

cd /usr/games/SteamCMD || exit 1
/usr/local/bin/sudo -u steamuser ./steamcmd.sh +login anonymous +app_update 222860 validate +quit

Running your game server does not change much from my previous post. The path of game folder is the only important modification. I've created a shell script to launch L4D2 server:

ROOT="/patpro/games/SteamApps/common/Left 4 Dead 2 Dedicated Server"
SUDO="/usr/local/bin/sudo -u steamuser"
NICE="/usr/bin/nice -n -5"
STEAMRUNARGS="-ip PUT-YOUR-IP-ADDRESS-HERE -fps_max 0 -sys_ticrate 1000"

cd "${ROOT}" || exit 1
${SCREEN} ${NICE} ${SUDO} ./srcds_run ${STEAMRUNARGS}

Open TCP and UDP ports 26901 and 27015 in your firewall, and edit SteamApps/common/Left 4 Dead 2 Dedicated Server/left4dead2/cfg/server.cfg to tweak your settings.

Happy gaming!

Track mpm-itk problems with truss

Some background

I've some security needs on a shared hosting web server at work and I've ended up installing Apache-mpm-itk in place of my old vanilla Apache server. MPM-ITK is a piece of software (a set of patches in fact) you apply onto Apache source code to change it's natural behavior.
Out of the box, Apache spawns a handful of children httpd belonging to user www:www or whatever your config is using. Only the parent httpd belongs to root.
Hence, every single httpd must be able to read (and sometimes write) web site files. Now imagine you exploit a vulnerability into a php CMS, successfully injecting a php shell. Now through this php shell, you are www on the server, you can do everything www can, and it's bad, because you can even hack the other web sites of the server that have no known vulnerability.
With MPM-ITK, Apache spawns a handfull of master processes running as root, and accordingly to your config files, each httpd serving a particular virtual host or directory will switch from root to a user:group of your choice. So, one httpd process currently serving files from web site "foo" cannot access file from web site "bar": an attacker exploiting a vulnerability of one particular web site won't be able to hack every other web site on the server.

More background

That's a world you could dream of. In real world, that's not so simple. In particular, you'll start having troubles as soon as you make use of fancy features, especially when you fail to provide a dedicated virtual host per user.
On the shared server we host about 35 vhosts for 250 web sites, and we can't afford to provide every user with his dedicated vhost. The result is a given virtual host with a default value for the fallback user:group (say www:www), and each web site configured via Directory to use a different dedicated user.

When a client GET a resource (web page, img, css...) it generally keeps the connection opened with the httpd process. And it can happen that a resource linked from a web page sits into another directory, belonging to another user. The httpd process has already switched from root to user1 to serve the web page, it can't switch to user2 to serve the linked image from user2's directory. So Apache drops the connection, spawns a new httpd process, switches to user2, and serves the requested resource.
When it happens, you can read things like this into your Apache error log:

[warn] (itkmpm: pid=38130 uid=1002, gid=80) itk_post_perdir_config(): 
initgroups(www, 80): Operation not permitted
[warn] Couldn't set uid/gid/priority, closing connection.

That's perfectly "legal" behavior, don't be afraid, unless you read hundreds of new warning every minute.
If you host various web sites, belonging to various users, into the same vhost, you're likely to see many of these triggered by the /favicon.ico request.

Where it just breaks

When things are getting ugly is the moment a user tries to use one of your available mod_auth* variant to add some user authentication (think .htaccess). Remember, I host many web sites in a single vhost, each one into its own directory with its own user:group.

Suddenly every single visitor trying to access the protected directory or subdirectory is disconnected. Their http client reports something like this:

the server unexpectedly dropped the connection...

and nothing else is available. The error, server-side, is the same initgroups error as above, and it does not help at all. How would you solve this? truss is your friend.

Where I fix it

One thing I love about FreeBSD is the availability of many powerful tools out of the box. When I need to track down a strange software behavior, I feel very comfortable on FreeBSD (it doesn't mean I'm skilled). truss is one of my favorites, it's simple, straightforward and powerful.
What you need to use truss is the PID of your target. With Apache + MPM-ITK, processes won't stay around very long, and you can't tell which one you will connect to in advance. So the first step is to buy yourself some precious seconds so that you can get the PID of your target before the httpd process dies. Remember, it dies as soon as the .htaccess file is parsed. Being in production, I could not just kill everything and play alone with the server, so I choose another way. I've created a php script that would run for few seconds before ending. Server side, I've prepared a shell command that would install the .htaccess file I need to test, and start truss while grabbing the PID of my target. On FreeBSD, something like this should do the trick:

cd /path/to/user1/web/site
mv .htaccess_inactive .htaccess && truss -p $(ps auxw|awk '/^user1/ {print $2}')

First http GET request, the .htaccess file is not present, an httpd process switches from root to user1, starts serving the php script. I launch my command server-side: it puts .htaccess in place, gets the PID of my httpd process, and starts truss.
The php script ends and returns its result, client-side I refresh immediately (second GET request), so that I stay on the same httpd process. My client is disconnected as soon as the httpd process has parsed the .htaccess file. At this point, truss should already be dead. I've the complete trace of the event. The best is to read the trace backward from the point where httpd process issue an error about changing UID or GID:

01: setgroups(0x3,0x80a8ff000,0x14,0x3,0x566bc0,0x32008) 
    ERR#1 'Operation not permitted'
02: getgid()					 = 80 (0x50)
03: getuid()					 = 8872 (0x22a8)
04: getpid()					 = 52942 (0xcece)
05: gettimeofday({1364591872.453335 },0x0)		 = 0 (0x0)
06: write(2,"[Fri Mar 29 22:17:52 2013] [warn"...,142) = 142 (0x8e)
07: gettimeofday({1364591872.453583 },0x0)		 = 0 (0x0)
08: write(2,"[Fri Mar 29 22:17:52 2013] [warn"...,85) = 85 (0x55)
09: gettimeofday({1364591872.453814 },0x0)		 = 0 (0x0)
10: shutdown(51,SHUT_WR)				 = 0 (0x0)

Line 01 is the one I'm looking for: the httpd process is trying to change groups and fails, line 02 to 05 it's gathering data for the log entry, line 06 it's writing the error to the log file. 07 & 08: same deal for the second line of log.

From that point in time, moving up shows that it tried to access an out-of-directory resource, and that resource is an html error page! Of course, it makes sense, and it's an hard slap on the head (RTFM!).

01: stat("/user/user1/public_html/bench.php",{ 
    mode=-rw-r--r-- ,inode=4121,size=7427,blksize=7680 }) = 0 (0x0)
02: open("/user/user1/public_html/.htaccess",0x100000,00) = 53 (0x35)
03: fstat(53,{ mode=-rw-r--r-- ,inode=4225,size=128,blksize=4096 }) = 0 (0x0)
04: read(53,"AuthType Basic\nAuthName "Admin "...,4096) = 128 (0x80)
05: read(53,0x80a8efd88,4096)			 = 0 (0x0)
06: close(53)					 = 0 (0x0)
07: open("/user/user1/public_html/bench.php/.htaccess",0x100000,00) 
    ERR#20 'Not a directory'
08: getuid()					 = 8872 (0x22a8)
09: getgid()					 = 80 (0x50)
10: stat("/usr/local/www/apache22/error/HTTP_UNAUTHORIZED.html.var",{ 
    mode=-rw-r--r-- ,inode=454787,size=13557,blksize=16384 }) = 0 (0x0)
11: lstat("/usr/local/www/apache22/error/HTTP_UNAUTHORIZED.html.var",{ 
    mode=-rw-r--r-- ,inode=454787,size=13557,blksize=16384 }) = 0 (0x0)
12: getuid()					 = 8872 (0x22a8)
13: setgid(0x50,0x805d43d94,0x64,0x800644767,0x101010101010101,0x808080808080
    8080) = 0 (0x0)

line 13 shows the beginning of setgid process, and 10/11 shows the culprit. Up from here is the regular processing of the .htaccess file.


When you use mod_auth* to present visitors with authentication, the server issues an error, and most of the time, this error is sent to the client with a dedicated header, and a dedicated html document (think "404"). When the error is about authentication (error 401), most clients hide the html part, and present the user with an authentication popup.
But the html part is almost always a physical file somewhere in the server directory tree. And it's this particular file the httpd process was trying to reach, issuing an initgroups command, and dying for not being allowed to switch users.
I've found in my Apache config the definition of ErrorDocument:

ErrorDocument 400 /error/HTTP_BAD_REQUEST.html.var
ErrorDocument 401 /error/HTTP_UNAUTHORIZED.html.var
ErrorDocument 403 /error/HTTP_FORBIDDEN.html.var

and replaced them all by a file-less equivalent, so Apache won't have any error file to read and will just send a plain ASCII error body (it saves bandwidth too):

ErrorDocument 400 "400 HTTP_BAD_REQUEST"
ErrorDocument 401 "401 HTTP_UNAUTHORIZED"
ErrorDocument 403 "403 HTTP_FORBIDDEN"

I've restarted Apache, and authentication from mod_auth* started to work as usual.
Same approach applies to almost any mpm-itk problem when it's related to a connection loss with Couldn't set uid/gid/priority, closing connection error log. You locate the resource that makes your server fail, and you find a way to fix the issue.

Benchmark: virtualized OS X vs Windows

Lately I've discussed the performance drop between a virtualized Mac OS X and the same system running natively on a Mac Pro. My virtualization project is not limited to Mac OS X of course. Windows, Linux, FreeBSD are also part of the deal. In order to further test my virtualized workstation setup, I've created a Windows Server 2008 R2 VM.
Every VM runs on top of ESXi, only one VM at a time so no interference is possible. Each VM uses the ATI Radeon HD 5770 PCIe card directly thanks to VMware passthrough mode. ESXi is running on a Mac Pro, and the native OS X system runs on the same Mac Pro so I have a consistent hardware platform.

I've given Cinebench a ride on this Windows VM, and I must admit, results are appalling… for Mac OS X:

Cinebench OS X 10.8.2 native OS X 10.8.2 VM Windows Server 2008 R2 VM
CORES 4 4 4
MHZ 2800 2663 2800
CBCPUX 5.038354 3.797552 3.962436
CBOPENGL 32.284100 27.319487 53.606468

I'm afraid a virtualized Windows system achieves better results than a native OS X. And not just a little bit better, but 66% better. We knew for ages that Apple ships crappy graphics card drivers and almost obsolete OpenGL. This is one more evidence.

After further research, I've finally succeeded in launching some Valve games on this windows VM: Half Life Lost Coast and Portal. They both run quite nicely. The HL Lost Coast integrated benchmark scores a very nice 229,82 FPS and the portal frame rate displayed by the command cl_showfps 1 was around 200 and 300.
On Team Fortress 2 I've been able to make a proper benchmark. That's not as detailed as my L4D2 bench, but that's enough.
I've recorded a game on TF2, Mac OS X 10.6.8, played it back with the timedemo command on the same system, and on the Windows VM.
It's a short demo (4099 frames) featuring a control point map with 12 players (11 bots, and me). Video settings were the same on both sides, of course.

Mac OS X 10.6.8 Native Windows VM
average 59.04 fps 59.83 fps
variability 2.764 fps 3.270 fps

It looks like something is capping the fps at 60. I don't know if it comes from my settings, or if it comes from outside the game. Both scores are very similar. Mac OS X's only bonus is the smaller variability, meaning its frame rate is more consistent throughout the demo. If only I had sound in my VMs…

Next step: try to configure a Ubuntu VM so it can use the ATI Radeon HD 5770 PCIe card, and make good use of my Steam On Linux beta test account.

Mac OS X on VMware ESXi: hardware challenges

I've decided to try and build a virtualized workstation that would allow me to use multiple OSes on top of my Mac Pro. That's no piece of cake, because it mainly boils down to using a professional hypervisor optimized for hardware abstraction and headless operation as a power-user workstation with full hardware access and as much GPU power as possible. It does not look like something that has a bright future, does it?

After some experiment I have a pretty good idea of what is possible and what is not possible. Lets compare the Mac Pro's hardware and what you can access from within a virtual machine running in ESXi on the same Mac Pro:

Mac Pro VM
CPU Full power with HT No HT, number of cores depends on the VM setup, but frequency can be lower than expected.
Running OS X 10.8 I got 2.66 GHz in the VM despite the 2.8 GHz Xeon
RAM Full RAM Depends on the VM setup, but if you use device passthrough, you must reserve the full amount of RAM, meaning you lose the ability to share unused RAM with other VMs. If you are a virtualization expert you know it's not good.
SATA Full access Possibility of raw device mapping
USB Full access, plug & play Passthrough available but limited: no keyboard and no mouse. Probably no plug & play either. Tested with logitech headphones: flaky sound with kernel log message complaining about a problem in USB driver, any app (itunes, chrome...) won't play sound any longer than 2 or 3 seconds before shutting down the sound output.
Bluetooth Full access none pseudo-passthrough available via USB devices, not tested.
Wifi Full access Passthrough possible, but not tested
LAN Full access Passthrough possible, not tested. Otherwise access via the virtual network stack of the hypervisor, works well.
Firewire Full access, plug & play none.
Graphics card Full access Passthrough possible, with a performance drop.
Some softwares will just not work, see last part of this post for details.
DVD Full access Passthrough possible, not tested.
Access via VCenter possible.
Optical sound output Full access none Passthrough of the Intel HD sound controler possible, but playback is out of sync, and so flaky it's unusable. On windows the sound device application commits suicide, on OS X the sound output is not even available.

This chart means important things. Running a virtualized Mac OS X workstation on top of ESXi will prevent me from:

  • using 100% CPU power (not that important)
  • using 100% of my RAM (not that important)
  • using 100% of my already limited GPU power (kind of important)
  • plug in USB devices like thumbdrives (important)
  • plug in Firewire devices like my CF-card reader (important)
  • accessing bluetooth device (I don't care)
  • using my optical Edirol MA-15D or any other good speaker (important)

Lets face it, those limitations alone could bring my project to a halt. I don't want a crappy workstation, and if virtualization is not the way to go, I might go the other way around and buy a small PC for every other OS I want to run. Even if it defeats the all-in-one purpose of the virtualization, it would allow me full access to each hardware resources.

Below, the "About this Mac" dialog featuring the VM on top and the real Mac Pro under.

comparison of about this mac dialog between OS X VM, and OS X running on the Mac Pro

Even simple hardware features are not well recognized, but it's enough for the average user experience. The GPU passthrough allows decent full screen 1080p HD video playback from youtube, and many games should work too. Unfortunately Valve's games won't work (Left 4 Dead…) as they make use of some framework that fails on Virtualized hardware.

hl2_osx[772]: -[__NSCFString bytes]: unrecognized selector sent to instance 0x2827350
hl2_osx[772]: An uncaught exception was raised
hl2_osx[772]: -[__NSCFString bytes]: unrecognized selector sent to instance 0x2827350
hl2_osx[772]: (
 0   CoreFoundation      0x988b212b __raiseError + 219
 1   libobjc.A.dylib     0x9545352e objc_exception_throw + 230
 2   CoreFoundation      0x988b5d9d -[NSObject(NSObject) doesNotRecognizeSelector:] + 253
 3   CoreFoundation      0x987fe437 ___forwarding___ + 487
 4   CoreFoundation      0x987fe1e2 _CF_forwarding_prep_0 + 50
 5   CoreFoundation      0x9878d720 CFDataGetBytePtr + 80
 6   launcher.dylib      0x0041c955 _ZN12GLMDisplayDB17PopulateRenderersEv + 2005
 7   launcher.dylib      0x00418607 _ZN12GLMDisplayDB8PopulateEv + 23
 8   launcher.dylib      0x0041b18f _ZN9CCocoaMgr12GetDisplayDBEv + 159
 9   shaderapidx9.dylib  0x0b28fb47 _ZN10IDirect3D921GetAdapterDisplayModeEjP15_D3DDISPLAYMODE + 55
 10  shaderapidx9.dylib  0x0b2db946 _ZNK19CShaderDeviceMgrDx818GetCurrentModeInfoEP19ShaderDisplayMode_ti + 38
 11  engine.dylib        0x05dbdeaa _Z14Shader_Connectb + 122
 12  engine.dylib        0x05f1232a _ZN10CEngineAPI7ConnectEPFPvPKcPiE + 106
 13  launcher.dylib      0x004151c3 _ZN15CAppSystemGroup9OnStartupEv + 115
 14  launcher.dylib      0x00415575 _ZN15CAppSystemGroup3RunEv + 37
 15  launcher.dylib      0x00415598 _ZN15CAppSystemGroup3RunEv + 72
 16  launcher.dylib      0x0041d202 _Z18MainFunctionThreadPv + 82
 17  launcher.dylib      0x0041d56c ValveCocoaMain + 140
 18  launcher.dylib      0x0040ca61 LauncherMain + 673
 19  hl2_osx             0x00001d26 start + 54

I've discovered that the [__NSCFString bytes]: unrecognized selector sent to instance error affects also Hackintosh users, ie. people running Mac OS X on top of non-Apple hardware.

Next step: try Valves games on the Windows VM, with GPU passthrough.

Mac OS X Benchmark: native vs virtualized, part 2

I've been really disappointed by my last benchmark of a virtualized Mac OS X running on top of ESXi with graphics card accessed in passthrough mode. So disappointed in fact that I had to make new tests.
This time, I've decided to ditch the six years old XBench, and to use proper video benchmarking tools: Geeks3D GpuTest, and Cinebench. And guess what? Thats better.
To run those tests, I've had to install OS X 10.8.2 because Geeks3D GpuTest doesn't run on Mac OS X 10.6.8. So I dedicated a SATA HDD on my Mac Pro to a fresh install of 10.8.2, created a VM with it and ran both benchmarks, once from the Mac Pro booted from OS X, once from the OS X VM.

In the chart bellow you can find FurMark and GiMark tests results for a native OS X system running on the Mac Pro, and for the exact same system running as a VM on top of ESXi hypervisor. No tuning was done, I've used the default settings for every benchmarks.

Geeks3D GpuTest Native VM
FurMark (AvgFPS / Score) 47 / 2845 47 / 2872
GiMark (AvgFPS / Score) 33 / 2000 7 / 446

FurMark scores the same frame rate on VM and on native OS X. But GiMark is not good at all, with a VM score 4.5x lower than reference.

Cinebench's results are quite interesting too:

Cinebench Native VM
MHZ 2800 2663
CBCPUX 5.038354 3.797552
CBOPENGL 32.284100 27.319487

VM results are quite close from reference, but the CPU frequency is reported as 2.663 GHz instead of 2.8 GHz, and the VM has only 4 CPU threads, instead of 8. This explain the CPU performance drop between native and virtualized OS X. The OpenGL score is quite good, showing only a 15.4% drop.
We are very far from the 87% drop on XBench's OpenGL test.

On the left side the native OS X, on the right side the virtualized OS X:
Cinebench results for OS X 10.8.2 native vs virtualized

Mac OS X Benchmark: native vs virtualized

An important thing about my work-in-progress virtualized workstation setup is that I've created the Mac OS X VM using my very own hard drives, hooked as raw devices (RDM: raw device mapping). So I can boot exactly the same OS directly from the hard drive, or from ESXi into a virtual machine. Quite convenient when the time comes to make comparisons. And now, I can boot the VM with ATI Radeon graphics card plugged in passthrough mode thanks to VMware DirectPath I/O and some tweaking.
While it's not enough to make a workstation (still miss a keyboard/mouse in passthrough), it allows some benchmarks. I've ran XBench on the VM and on the same OS booted natively from the hard drive.

The VM is configured with only 4 CPU. the Mac Pro sports a quad core Xeon capable of hyperthreading, so when Mac OS X boots natively it sees 8 CPU. It might explain the 50% difference on the Thread test, but that will require further testing.

The final result is not good at all. I understand very well that virtualization has a performance cost, but if I want a powerful virtualized workstation I need a setup that will waste as few resources as possible.
Quartz Graphics and User Interface tests show that "desktop" graphics are well supported, but the OpenGL test results are horrendous. With a performance loss of 87%, it predicts much trouble with games. According to this very simple benchmark, the VMware passthrough mode for graphics card seems to be very bad compared to what can be seen on XEN for example.
To be honest, having my hard disks accessed directly via RDM, I though I would have a 10-15% penalty. The 46% drop for sequential access surprises me. As for the GPU, the OpenGL results are so bad I'm wondering if the graphics card is properly passed through. May be some features are just dropped in the process. By the way, the virtualized Mac OS X won't load the screen color profile. May be it's related to the pseudo-VGA screen attached to the VSphere console. Unfortunately I can't get rid of this pseudo-VGA screen yet. Until I find a way to pass keyboard and mouse through to the VM, I need the VSphere console.

Results 259,39 127,18 -50,97 %
System Info
Xbench Version 1,3 1,3
System Version 10,6,8 (10K549) 10,6,8 (10K549)
Physical RAM 24576 MB 12288 MB
Model MacPro5,1 VMware7,1
Drive Type WDC WD1001FALS WDC WD1001FALS (ATA)
CPU Test 205,42 200,27 -2,51 %
GCD Loop 314,75 16,59 Mops/s 305,66 16,11 Mops/s -2,89 %
Floating Point Basic 182,81 4,34 Gflop/s 177,44 4,22 Gflop/s -2,94 %
vecLib FFT 121,5 4,01 Gflop/s 119,14 3,93 Gflop/s -1,94 %
Floating Point Library 385,38 67,11 Mops/s 374,17 65,15 Mops/s -2,91 %
Thread Test 954,74 477,33 -50,00 %
Computation, 4 thr. 989,65 20,05 Mops/s 517,69 10,49 Mops/s -47,69 %
Lock Contention, 4 thr. 922,22 39,67 Mlocks/s 442,8 19,05 Mlocks/s -51,99 %
Memory Test 452,72 370,19 -18,23 %
System 493,4 452,74 -8,24 %
Allocate 746,78 2,74 Malloc/s 877,63 3,22 Malloc/s 17,52 %
Fill 352,03 17116,62 MB/s 287,47 13977,29 MB/s -18,34 %
Copy 526,18 10867,96 MB/s 497,95 10285,00 MB/s -5,37 %
Stream 418,24 313,1 -25,14 %
Copy 422,51 8726,77 MB/s 321,23 6634,92 MB/s -23,97 %
Scale 395,84 8178,02 MB/s 303,88 6278,02 MB/s -23,23 %
Add 438,89 9349,24 MB/s 328,71 7002,18 MB/s -25,10 %
Triad 417,99 8941,89 MB/s 300,37 6425,59 MB/s -28,14 %
Quartz Graphics Test 315,19 300,47 -4,67 %
Line [50% α] 239,24 15,93 Klines/s 232,29 15,47 Klines/s -2,91 %
Rectangle [50% α] 314,61 93,93 Krects/s 296,25 88,45 Krects/s -5,84 %
Circle [50% α] 264,41 21,55 Kcircles/s 251,89 20,53 Kcircles/s -4,74 %
Bezier [50% α] 279,29 7,04 Kbeziers/s 263,55 6,65 Kbeziers/s -5,64 %
Text 875,44 54,76 Kchars/s 836,28 52,31 Kchars/s -4,47 %
OpenGL Graphics Test 306,01 39,01 -87,25 %
Spinning Squares 306,01 388,19 frames/s 39,01 49,49 frames/s -87,25 %
User Interface Test 463,72 405,19 -12,62 %
Elements 463,72 2,13 Krefresh/s 405,19 1,86 Krefresh/s -12,62 %
Disk Test 97,42 72,35 -25,73 %
Sequential 176,21 94,27 -46,50 %
Uncached Write [4K blk.] 180,38 110,75 MB/s 167,14 102,62 MB/s -7,34 %
Uncached Write [256K blk.] 177,43 100,39 MB/s 80,84 45,74 MB/s -54,44 %
Uncached Read [4K blk.] 149,41 43,73 MB/s 51,88 15,18 MB/s -65,28 %
Uncached Read [256K blk.] 207,16 104,12 MB/s 208,09 104,59 MB/s 0,45 %
Random 67,32 58,7 -12,80 %
Uncached Write [4K blk.] 21,3 2,25 MB/s 19,86 2,10 MB/s -6,76 %
Uncached Write [256K blk.] 507,04 162,32 MB/s 300,75 96,28 MB/s -40,69 %
Uncached Read [4K blk.] 159,73 1,13 MB/s 96,75 0,69 MB/s -39,43 %
Uncached Read [256K blk.] 235,97 43,79 MB/s 242,2 44,94 MB/s 2,64 %