Escaping the Apple ecosystem: a view of the setup

Here is a quick & dirty view of the physical and logical setup of my new workstation. The linux part is not finished yet (no drivers for Radeon GPU, thank you Ubuntu), it's a work in progress.

esx
Not depicted: each USB controller sports 4 USB ports (yellow) or 2 USB ports (pink and blue). It allows me to plug few devices that won't be "managed" by the USB switch.
USB devices plugged-in on the switch are made available to only one VM at a time. When I press the switch button, they disappear for the current VM and are presented to the next one.

Escaping the Apple ecosystem: part 3

In part 2, I was able to create and use a Windows 7 VM with the Radeon R9 270x in passthrough. It works really great. But OSX and Linux where more difficult to play with.

List of virtual machines

List of virtual machines


Since then, I've made tremendous progress: I've managed to run an OSX 10.11.6 VM properly, but more importantly, I've managed to run my native Mac OS X 10.6.8 system as a VM, with the Mac's Radeon in passthrough.
I've removed my Mac OS X SSD and the Mac's graphics card from the Mac Pro tower, and installed them into the PC tower. Then I've created the VM for the 10.6.8 system, configured ESXi to use Mac's Radeon with VT-d, etc.
The only real problem here is that adding a PCI card into the PC tower makes PCI device numbers change: it breaks almost every passthrough already configured. I had to remake VT-d config for the Windows VM. Apart from that, it went smoothly.
Currently, I'm working on my native 10.6.8 system, that runs as a VM, and the Windows VM is playing my music (because the Realtek HD audio controller is dedicated to the Windows VM).
Moving from a Mac Pro with 4-core 2.8 GHz Xeon to a 6-core 3.5 GHz Core i7 really gives a boost to my old 10.6.8 system.

Running both OSes, the box is almost as silent as the Mac Pro while packing almost twice as more raw CPU power and 2.7x more GPU power.

The Mac Pro is now empty: no disks, no graphics card, and will probably go on sale soon.

to-do list:

  • secure the whole infrastructure ;
  • install 2nd-hand MSI R9 270x when it's delivered ;
  • properly setup Linux to use AMD graphics card.

I might also add few SSDs and a DVD burner before year's end.

Escaping the Apple ecosystem: part 2

In part 1, I've written about the BoM of my project and the associated to-do list.
First item on this list was: build the box. That did not go as smoothly as expected. The motherboard was not fully operational: after few minutes of run time (between 5 and 30), it would trigger a CPU overheat alarm, even when the CPU was idle and cool. Supermicro's Support made me tried a new BIOS, with no effect, so I've finally send the board for exchange. The new board arrived but I've had to delay the rebuilt for few weeks.
Now the PC is up and running. The new motherboard seems to work great, but I've not tested IPMI yet. IPMI was the very first feature I've used on the first board, and there is a slight probability that the CPU overheat problem comes from a probe malfunction related to the BMC. Let's keep that for later.

I've chosen to run this box on VMware ESXi 5.5, because it's quite common (more than the latest 6.x), because it sports features I need like passthrough, and because most VMware based multi-sit PC projects like mine are using ESXi 5.x.

ESXi is quite easy to install, I won't give details. Main hdd in the box is installed with ESXi, which is configured thanks to a USB keyboard and a display plugged in the VGA port of the motherboard. After basic configuration (network, user password...), I've switched to remote configuration through VSphere Client, installed on a Windows PC (really a VM running on the Mac).

General view

General view of ESX's configuration


Configuring passthrough for GPUs is pretty straightforward, because I've started with only one GPU, and because these are discrete PCI cards. On the other hand, passthrough of USB controller can be tricky: many controllers, nothing to identify them except trial & error (unless you have the blueprint for your motherboard telling what physical USB ports belong to what controller).
Go to configuration tab, then "Advanced", and finally click "edit" on the right

Go to configuration tab, then "Advanced", and finally click "edit" on the right


When you click "Edit…" a window opens that lists interesting devices you can try to passthrough.
Choose some. Then you have to reboot the ESXi and add some of these devices to a VM.
passthrough
I've created a Windows 7 pro 64bits VM, with raw device mapping pointing to an SSD. I've added every available PCI devices to this VM (USB, sound, GPU) and installed Windows plus updates.
It's important to remember that initially, most PCI devices might not work at all because of missing drivers on guest OS (here it's Windows). Hence, after installing Windows, the Radeon was detected but not used, and only Intel USB controllers where working. I've installed AMD drivers, and ASMedia drivers (courtesy of Supermicro). I've also installed VMware Tools.
After all this, the Windows VM properly uses my Dell display hocked-up on the MSI R9 270x Radeon, and I can interact with the system thanks to a real keyboard and mouse. Passing through the whole USB controller allows me to use any USB device I want. I've successfully plugged-in and used a thumbdrive, a USB gaming headset and a USB hub.
I've made some GPU/CPU benchmarks and everything looks perfect. I've tested Left 4 Dead 2 game play, and it looked great too (I'll probably have to tweak anti aliasing settings to make it perfect).

The Windows part was quite fast to setup, and is almost done now. I've started to fight with OSX and Ubuntu, but things are not easy with both of them. It looks like my 3 years old graphics card it so new that OSX does not support it until 10.11.x, and Ubuntu won't allow me to install Radeon drivers on 16.x LTS because they wait for some software to stabilize before packaging it…

To-do list:

  • fix problems with OSX and Ubuntu virtualization
  • find another MSI Radeon R9 270X GAMING 2G (of course it's no longer in stock…)
  • fully test Mac OS X 10.6.8 with Mac's graphics card instead of MSI Radeon

Escaping the Apple ecosystem: part 1

X10SRA-F_specBack in late 2012, I've started to think about my post-Apple days. I knew already that I would not endorse the full cloud crap, and the oversimplification of OS X that follows the iOS convergence. My hardware, my OS, my data.

I'm still running a Mac Pro 2010, with Mac OS X 10.6.8, the last great OS from Apple. Unfortunately (in)security is not what it used to be, and browsers, ssl libraries, etc. are updated frequently and older OSes are no longer supported. So it was time to switch away from 10.6.8 for my online activities. Meaning, it's time to create my multi-headed workstation running various OSes with dedicated GPU.

After about a year spent looking for the right pieces of hardware, I've came up with this BOM:

Hopefully it will yield to great results with virtualization, passthrough of GPUs and USB.
For now, only one GPU, just in case it's not working accordingly. That will allow me to test every OSes I want to use with GPU passthrough. I'll buy 2 more GPUs when it's fully tested and satisfactory.

To do list:

  • Assemble the workstation
  • Install and patch VMWare (to allow the virtualization of Mac OS X on non-Apple hardware)
  • Test each VM with GPU+USB+Sound passthrough

Due to the exotic nature of some of these pieces of hardware, I've had to order them from four retailers in two countries: Amazon (France), LDLC (France), PC-Cooling (Germany) and MindFactory (Germany).

I've got only one delivery problem, for the PC case: it was delivered through GLS Group, and it took them more than a week to drive 10 km from their warehouse to my home. When it finally went through my door, the box was trashed and the PC case had one foot slightly hammered into the case. It can't stand still on its 4 feet.
Whatever you buy abroad, just make sure it won't be delivered through GLS. The driver was a pain in the ass, calling me to delay the delivery.

Le PC dans mon téléphone

Précédemment, j'ai évoqué l'ordinateur de demain. J'ai fait un test grandeur nature façon corporate. Comprendre : vous aurez du mal à faire pareil chez vous. J'utilise en effet quelques artifices comme un accès VPN au réseau de mon employeur, et au travers de cet accès une machine virtuelle VMware View.
Au final, j'ai le bureau d'une machine Windows 7 affiché sur la TV du salon, j'ai un clavier et une souris bluetooth, le tout servi par mon téléphone malin, via un tunnel VPN dans une connexion WIFI. La connexion avec la machine virtuelle se fait via le client officiel View, de VMware (donc en PCoIP).

Windows dans ma TV sur mon téléphone

Ci dessus une photo de la TV affichant le bureau de la VM dans laquelle est lancé le client VSphere (outil d'administration de datacenter VMware).

À 2m50, le texte dans l'interface n'est pas hyper lisible sur l"écran 80cm, donc pour le PC dans le canapé, on repassera, ou alors on fera des réglages spéciaux sur la machine virtuelle.

Le client View se comporte relativement bien, néanmoins, il y a comme un petit souci avec le clavier. Ce dernier fonctionne parfaitement en azerty partout, sauf à l'intérieur du Windows, où il fonctionne en qwerty. Le client View dispose d'un champs de saisie qui permet de taper du texte à l'extérieur de la VM et de l'envoyer ensuite vers cette dernière. À cet endroit, c'est bien de l'azerty. Mais le même texte tapé directement dans la VM passe en qwerty. C'est peut être du à mon téléphone, qui est paramétré tout en anglais.

Aussi, suite à un blocage logiciel sur ma VM, j'ai tenté un ctrl-alt-suppr via mon clavier bluetooth. Le programme n'a pas rendu la main, la VM n'a pas rebooté. Non. C'est le téléphone qui a rebooté. Ça m'a fait rire.

Benchmark: virtualized OS X vs Windows

Lately I've discussed the performance drop between a virtualized Mac OS X and the same system running natively on a Mac Pro. My virtualization project is not limited to Mac OS X of course. Windows, Linux, FreeBSD are also part of the deal. In order to further test my virtualized workstation setup, I've created a Windows Server 2008 R2 VM.
Every VM runs on top of ESXi, only one VM at a time so no interference is possible. Each VM uses the ATI Radeon HD 5770 PCIe card directly thanks to VMware passthrough mode. ESXi is running on a Mac Pro, and the native OS X system runs on the same Mac Pro so I have a consistent hardware platform.

I've given Cinebench a ride on this Windows VM, and I must admit, results are appalling… for Mac OS X:

Cinebench OS X 10.8.2 native OS X 10.8.2 VM Windows Server 2008 R2 VM
CORES 4 4 4
LOGICALCORES 2 1 1
MHZ 2800 2663 2800
CBCPUX 5.038354 3.797552 3.962436
CBOPENGL 32.284100 27.319487 53.606468

I'm afraid a virtualized Windows system achieves better results than a native OS X. And not just a little bit better, but 66% better. We knew for ages that Apple ships crappy graphics card drivers and almost obsolete OpenGL. This is one more evidence.

After further research, I've finally succeeded in launching some Valve games on this windows VM: Half Life Lost Coast and Portal. They both run quite nicely. The HL Lost Coast integrated benchmark scores a very nice 229,82 FPS and the portal frame rate displayed by the command cl_showfps 1 was around 200 and 300.
On Team Fortress 2 I've been able to make a proper benchmark. That's not as detailed as my L4D2 bench, but that's enough.
I've recorded a game on TF2, Mac OS X 10.6.8, played it back with the timedemo command on the same system, and on the Windows VM.
It's a short demo (4099 frames) featuring a control point map with 12 players (11 bots, and me). Video settings were the same on both sides, of course.

Mac OS X 10.6.8 Native Windows VM
average 59.04 fps 59.83 fps
variability 2.764 fps 3.270 fps

It looks like something is capping the fps at 60. I don't know if it comes from my settings, or if it comes from outside the game. Both scores are very similar. Mac OS X's only bonus is the smaller variability, meaning its frame rate is more consistent throughout the demo. If only I had sound in my VMs…

Next step: try to configure a Ubuntu VM so it can use the ATI Radeon HD 5770 PCIe card, and make good use of my Steam On Linux beta test account.

Mac OS X on VMware ESXi: hardware challenges

I've decided to try and build a virtualized workstation that would allow me to use multiple OSes on top of my Mac Pro. That's no piece of cake, because it mainly boils down to using a professional hypervisor optimized for hardware abstraction and headless operation as a power-user workstation with full hardware access and as much GPU power as possible. It does not look like something that has a bright future, does it?

After some experiment I have a pretty good idea of what is possible and what is not possible. Lets compare the Mac Pro's hardware and what you can access from within a virtual machine running in ESXi on the same Mac Pro:

Mac Pro VM
CPU Full power with HT No HT, number of cores depends on the VM setup, but frequency can be lower than expected.
Running OS X 10.8 I got 2.66 GHz in the VM despite the 2.8 GHz Xeon
RAM Full RAM Depends on the VM setup, but if you use device passthrough, you must reserve the full amount of RAM, meaning you lose the ability to share unused RAM with other VMs. If you are a virtualization expert you know it's not good.
SATA Full access Possibility of raw device mapping
USB Full access, plug & play Passthrough available but limited: no keyboard and no mouse. Probably no plug & play either. Tested with logitech headphones: flaky sound with kernel log message complaining about a problem in USB driver, any app (itunes, chrome...) won't play sound any longer than 2 or 3 seconds before shutting down the sound output.
Bluetooth Full access none pseudo-passthrough available via USB devices, not tested.
Wifi Full access Passthrough possible, but not tested
LAN Full access Passthrough possible, not tested. Otherwise access via the virtual network stack of the hypervisor, works well.
Firewire Full access, plug & play none.
Graphics card Full access Passthrough possible, with a performance drop.
Some softwares will just not work, see last part of this post for details.
DVD Full access Passthrough possible, not tested.
Access via VCenter possible.
Optical sound output Full access none Passthrough of the Intel HD sound controler possible, but playback is out of sync, and so flaky it's unusable. On windows the sound device application commits suicide, on OS X the sound output is not even available.

This chart means important things. Running a virtualized Mac OS X workstation on top of ESXi will prevent me from:

  • using 100% CPU power (not that important)
  • using 100% of my RAM (not that important)
  • using 100% of my already limited GPU power (kind of important)
  • plug in USB devices like thumbdrives (important)
  • plug in Firewire devices like my CF-card reader (important)
  • accessing bluetooth device (I don't care)
  • using my optical Edirol MA-15D or any other good speaker (important)

Lets face it, those limitations alone could bring my project to a halt. I don't want a crappy workstation, and if virtualization is not the way to go, I might go the other way around and buy a small PC for every other OS I want to run. Even if it defeats the all-in-one purpose of the virtualization, it would allow me full access to each hardware resources.

Below, the "About this Mac" dialog featuring the VM on top and the real Mac Pro under.

comparison of about this mac dialog between OS X VM, and OS X running on the Mac Pro

Even simple hardware features are not well recognized, but it's enough for the average user experience. The GPU passthrough allows decent full screen 1080p HD video playback from youtube, and many games should work too. Unfortunately Valve's games won't work (Left 4 Dead…) as they make use of some framework that fails on Virtualized hardware.

hl2_osx[772]: -[__NSCFString bytes]: unrecognized selector sent to instance 0x2827350
hl2_osx[772]: An uncaught exception was raised
hl2_osx[772]: -[__NSCFString bytes]: unrecognized selector sent to instance 0x2827350
hl2_osx[772]: (
 0   CoreFoundation      0x988b212b __raiseError + 219
 1   libobjc.A.dylib     0x9545352e objc_exception_throw + 230
 2   CoreFoundation      0x988b5d9d -[NSObject(NSObject) doesNotRecognizeSelector:] + 253
 3   CoreFoundation      0x987fe437 ___forwarding___ + 487
 4   CoreFoundation      0x987fe1e2 _CF_forwarding_prep_0 + 50
 5   CoreFoundation      0x9878d720 CFDataGetBytePtr + 80
 6   launcher.dylib      0x0041c955 _ZN12GLMDisplayDB17PopulateRenderersEv + 2005
 7   launcher.dylib      0x00418607 _ZN12GLMDisplayDB8PopulateEv + 23
 8   launcher.dylib      0x0041b18f _ZN9CCocoaMgr12GetDisplayDBEv + 159
 9   shaderapidx9.dylib  0x0b28fb47 _ZN10IDirect3D921GetAdapterDisplayModeEjP15_D3DDISPLAYMODE + 55
 10  shaderapidx9.dylib  0x0b2db946 _ZNK19CShaderDeviceMgrDx818GetCurrentModeInfoEP19ShaderDisplayMode_ti + 38
 11  engine.dylib        0x05dbdeaa _Z14Shader_Connectb + 122
 12  engine.dylib        0x05f1232a _ZN10CEngineAPI7ConnectEPFPvPKcPiE + 106
 13  launcher.dylib      0x004151c3 _ZN15CAppSystemGroup9OnStartupEv + 115
 14  launcher.dylib      0x00415575 _ZN15CAppSystemGroup3RunEv + 37
 15  launcher.dylib      0x00415598 _ZN15CAppSystemGroup3RunEv + 72
 16  launcher.dylib      0x0041d202 _Z18MainFunctionThreadPv + 82
 17  launcher.dylib      0x0041d56c ValveCocoaMain + 140
 18  launcher.dylib      0x0040ca61 LauncherMain + 673
 19  hl2_osx             0x00001d26 start + 54
 )

I've discovered that the [__NSCFString bytes]: unrecognized selector sent to instance error affects also Hackintosh users, ie. people running Mac OS X on top of non-Apple hardware.

Next step: try Valves games on the Windows VM, with GPU passthrough.

Mac OS X Benchmark: native vs virtualized, part 2

I've been really disappointed by my last benchmark of a virtualized Mac OS X running on top of ESXi with graphics card accessed in passthrough mode. So disappointed in fact that I had to make new tests.
This time, I've decided to ditch the six years old XBench, and to use proper video benchmarking tools: Geeks3D GpuTest, and Cinebench. And guess what? Thats better.
To run those tests, I've had to install OS X 10.8.2 because Geeks3D GpuTest doesn't run on Mac OS X 10.6.8. So I dedicated a SATA HDD on my Mac Pro to a fresh install of 10.8.2, created a VM with it and ran both benchmarks, once from the Mac Pro booted from OS X, once from the OS X VM.

In the chart bellow you can find FurMark and GiMark tests results for a native OS X system running on the Mac Pro, and for the exact same system running as a VM on top of ESXi hypervisor. No tuning was done, I've used the default settings for every benchmarks.

Geeks3D GpuTest Native VM
FurMark (AvgFPS / Score) 47 / 2845 47 / 2872
GiMark (AvgFPS / Score) 33 / 2000 7 / 446

FurMark scores the same frame rate on VM and on native OS X. But GiMark is not good at all, with a VM score 4.5x lower than reference.

Cinebench's results are quite interesting too:

Cinebench Native VM
CORES 4 4
LOGICALCORES 2 1
MHZ 2800 2663
CBCPUX 5.038354 3.797552
CBOPENGL 32.284100 27.319487

VM results are quite close from reference, but the CPU frequency is reported as 2.663 GHz instead of 2.8 GHz, and the VM has only 4 CPU threads, instead of 8. This explain the CPU performance drop between native and virtualized OS X. The OpenGL score is quite good, showing only a 15.4% drop.
We are very far from the 87% drop on XBench's OpenGL test.

On the left side the native OS X, on the right side the virtualized OS X:
Cinebench results for OS X 10.8.2 native vs virtualized