cdelorme at gmail
May 15, 2012, 9:35 AM
Sorry for the delay, I meant to reply yesterday but had a lot of trouble
Re: gaming on multiple OS of the same machine?
[In reply to]
writing up an appropriate answer, as I am no specialist when it comes to
I have respect for anyone willing to invest in experimental technology, but
that said I would never recommend something if I didn't know it would work,
and work better than an existing alternative.
It is possible to get four HVM's with GPU Passthrough on one machine, but I
don't believe it is feasible. I would think it easier to try out a dual
setup first, especially if you are new to Xen, and if that works you can
reproduce it easily.
It would be a tragedy to invest in amazing hardware equivalent to the cost
of two machines only to encounter a fatal problem (whether mentioned here
or brand new).
I was new to Xen as of March, it took me two months of tinkering to get a
firm enough grasp of Xen that I managed GPU passthrough to one machine.
After a month more, I can safely say I could probably produce two to three
HVM's with passthrough on my current hardware if I wanted to go that route.
As Andrew stated, there is no "perfect" board, I looked just a couple of
months ago and failed and I looked at hundreds of boards from at least a
dozen manufacturers. It's clearly possible to setup a four machine
passthrough system, but I wouldn't consider it feasible.
If you want to save time, sanity, and possible an expensive investment, I
would stick with a more feasible setup and duplicate it for the same cost.
Given that Andrew has successfully setup what you are looking to, it would
be wise to take him up on his offer and get hardware & setup details.
Intel Core i7 processors have hyperthreading, a 4 core i7 is the equivalent
of an 8-core AMD, so if Andrew has had success with an 8-core AMD CPU and
no resource issues, you might be able to get by with a 4-core i7 (a 6 core
i7 would be 12 vcores), but I would be hesitant to assign all the cores to
Andrew is correct, stay away from NF200, it's not compatible. There are
certain required features it doesn't have, which can be checked with "lspci
-vvv", but I don't recall what the flags you would be looking for are.
PLX is a great product, I don't deny that in the least, and it is IOMMU
compatible which is awesome. However, I believe the flaw comes by design
or implementation of PLX on the board.
In my case the Extreme7 shared two PCIe slots with other components using
the PLX bridge. While NOT the case with my Extreme4, when I passed one of
those "slots" with the Extreme7 it treated the entire bridge as a single
PCI device and all components on it were passed via xen-pciback.hide. It
was an awful experience, I lost internet, various input devices, and due to
the lack of documentation on reversing xen-pciback.hide I was unable to fix
it without reinstalling.
I found lspci to be of little help identifying it, as a bridge it just acts
as a layer, so you can use tree view (lspci -tv) for details, but it won't
say whether the devices are all tied together. My guess is the PLX was
bridged to the PCIe slot, and they shared the same space, the tree on the
Extreme4 has tons of unused bridge ports in contrast to the Extreme4 which
was completely filled.
This experience also leads me to question how the Dual GPU cards will be
handled, as a single merged device or separately. Depending on PLX
"implementation" it could go either way.
In conclusion, if you get a board with PLX it will pass devices, but be
prepared (by which I mean keep regular backups as you progress) to hit a
wall or two during testing.
For five machines (4x Windows and Dom0) each to have a keyboard and mouse
you need 10 USB ports. If you choose to use PCI Passthrough you will also
need five USB PCI Buses.
My Extreme4 comes with four USB PCI Buses onboard (2x USB 3.0, and 2x. USB
2.0), so in that sense you might need a spare PCI slot for a USB Bus
If you choose to use USB Passthrough you won't need as many PCI Buses, but
you may not have the ability to pass complex devices that require drivers.
For example, I could not get an XBox 360 Wireless Adapter working with USB
Passthrough (PCI Passthrough it works fine).
On that same note, I encountered problems with USB 3.0, I believe they were
just USB incompatibilities and not Xen or PCI Passthrough related, you
won't have any trouble with normal input devices.
My board also has a Marvel SATA controller which did not work with VT-d
(DMA READ operation fails), so I had to disable it in UEFI to prevent
lengthy and verbose errors at boot time. For me loosing two SATA ports was
not a big deal, for others this could be a deal breaker.
While not a limitation per say, you may want to consider buying more hard
drives to run the virtual machines on, four machines making IO requests to
partitions on the same drive could get sluggish even with PV drivers.
In line with what Andrew said, manufacturers don't list the information you
would need to verify any of thus stuff, you essentially have to buy it and
try it, or look for other peoples reported successes.
In contrast to what Andrew has reported, my problem is OpenGL for
Accelerated "Graphics", oddly Accelerated Video works great. Youtube works
with the box checked just fine, and I play a large number of media files
and formats without any problems (except for h.264 10-bit video playback,
had to change to an alternative codec since Microsoft does not support
Direct3D works, I have played Borderlands and The Last Remnant at max
settings without lag. To test compatibility I also installed Final Fantasy
7 & 8 which work with related patch files. I am pretty sure the patch
files open the game and run it through an OpenGL filter of some kind, so I
have to imagine that works. While I am using an AMD/ATI card, Borderlands
installed nVidia PhysX which is also working.
The error messages I have encountered appear to be with specific
applications, more specifically anything that has to do with image editing
software. Photoshop and Blender 3D Sculpting software are the problem, I
think Blender uses a Java library. Photoshop runs but says no GPU
Acceleration, Blender doesn't open at all. Search results all say it's
because they can't be told to use a secondary graphics card.
When I searched for details about either message, they both led me to
problems with the software using a Secondary Graphics card. Perhaps I
should find it funny that games I can buy for $30 are more compatible with
secondary graphics than $400 professional image editing software.
Do keep us updated on your progress with the system, it's always nice to
have more people giving this a try.
On Tue, May 15, 2012 at 9:48 AM, Andrew Bobulsky <rulerof [at] gmail> wrote:
> Hello Peter,
> Answering some of your questions inline below:
> On Mon, May 14, 2012 at 5:52 PM, Peter Vandendriessche
> <peter.vandendriessche [at] gmail> wrote:
> > Hi Casey,
> > Thanks a lot for the bunch of information. Some further questions on the
> > and the VGA passthrough though.
> > On Sun, May 13, 2012 at 8:42 PM, Casey DeLorme <cdelorme [at] gmail>
> >> I checked every major manufacturers high end boards, searching for one
> >> with the best features to price and was aiming for the most PCIe slots I
> >> could get. Pretty much Every single board with more than 3 PCIe (x16)
> >> came with some form of PCI switch. Most of these PCI switches break
> >> in one way or another.
> > Yes, indeed, all of the 4-GPU-motherboards I know, have 2 PCI-e x16 slots
> > which split to 2 x8 each (so 4 x8 in total). Is this always a fatal
> > Is there any easy way to find out if it will be a problem (like from the
> > lspci info)?
> I couldn't say for sure if there's any great way to identify what
> would be a problem and what wouldn't. I don't know too much in the
> way of the nitty-gritty details, but the problem with NF200 PCIe
> switched boards, in particular, has something do with the fact that
> they route data to the southbridge, rather than the northbridge.
> Something about it makes it so that it's not quite "real" PCIe
> switching, it's more analogous to PCIe "routing," if that's a fair
> term to use. Instead of switching traffic on the existing PCIe bus
> that's natively part of the northbridge, they "bolt on" a second PCIe
> bus through some kind of trickery that prevents a solid line of
> "native PCIe" from being visible to the IOMMU. IIRC, the entire
> contents of the NF200 bus itself is a valid passthrough device, but
> the individual devices on it are not.... which kinda defeats the
> purpose, IMHO. This is all just my recollection of when I looked into
> it over a year ago, so I could be wrong about why it doesn't work...
> but Casey's advice is sound: stay away from NF200. :)
> Which brings me to PLX...
> >> The PLX was an entirely different form of problem, it merges device
> >> "function" with device identifiers. If you run "lspci" you get a list
> >> your devices, they are identified by "bus:device.function".
> >> [...]
> >> Armed with this knowledge, here is where you may run into problems:
> >> - Finding a board with 4x PCIe x16 Slots not tied to a PCI Switch
> > So, do I understand correct that it will work if and only if there is 1
> > "bus" per PCI-e slot?
> In my experience, PLX PCIe switches are *fantastic*. They speak
> native PCIe, and all the devices that they hook onto the PCIe bus have
> a clear logical path to the IOMMU. The PLX switch built into the
> Radeon 6990 is the reason you could attach each GPU to a different VM
> (though, as I said, the architecture of the card PAST that point is
> the reason this wouldn't be of any use to you), and the PLX chip on
> the HighPoint RU1144A is the reason that each port on the card can be
> assigned to a different VM. Granted, while the purpose of the RU1144A
> is to provide four full-bandwidth USB3 ports to a host machine, the
> way that HighPoint ended up architecting that goal provided me with
> exactly the piece of hardware I needed to work out my USB connectivity
> crisis, a purpose I highly doubt the engineers who designed it had
> even thought of... but I thank them nonetheless :)
> >> - Sparing enough USB ports for all machines input devices
> > What do you mean by sparing here? On a different bus than the PCI-e slots
> > for the GPUs?
> Specifically, what I think he's getting at is partly what I addressed
> above; In order to supply your VMs with USB, the best bet here is to
> use PCIe passthrough to hook different controllers to different VMs.
> It's highly likely that the architecture of your mainboard will
> prohibit pulling this off when you're going to have more than two
> different guest OSes which each need their own PCI -> USB adapter.
> Normally, because this kind of thing basically *never* matters,
> especially seeing as how cost-prohibitive it would be to facilitate it
> from the standpoint of motherboard manufacturers, you'll probably not
> be able to find a board that will allow you to work out the problem.
> Even if you did, I highly doubt it would have four 16x PCIe slots!
> Also, since this kind of information is damn near impossible to come
> by without getting the hardware in hand and checking it out
> yourself... it's probably a lost cause to look for the "perfect"
> >> - Limited to around 3GB of RAM per HVM unless you buy 8GB RAM Chips
> > Neither seems a problem (3GB RAM per machine or 8GB RAM chips). The
> price of
> > RAM is fairly linear in its size.
> I'll second your notion. 3.5 GB per DomU is more than enough for
> pretty much any game. It'd be a different story if you were
> virtualizing SQL databases or something, but games are designed to run
> on systems that would probably make most enthusiasts on Xen-Users list
> cry if they had to use such a machine on a daily basis :D
> >> - Will need a 6-core i7 to power all systems without potential resource
> >> conflicts
> > Okay. That rises the price a bit, but it'd still be well worth it.
> At LEAST a 6 core chip. If you build the AMD route, just shovel out
> for the 8 core chip instead, so you know you're golden. Dom0 won't be
> doing too much, and that leaves two cores for each VM that they can
> max out without even being capable of trampling eachother.
> >> - Encountering bugs nobody else has when you reach that 3rd or 4th HVM
> > This would probably be the real problem, given that I'm new to Xen.
> I've seen odd behavior myself. Sometimes, one of the VMs USB ports
> won't work. I switch the PCIe assignments around, and it's just a
> particular VM that's being fishy... nothing wrong with the actual
> controller. It's been weird, but I attribute it to the fact that my
> USB controller is plugged into a riser cable. I've seen devices get
> funny when you do that. However, it was the only way to plug in four
> GPUs AND a PCIe USB controller :P
> >> I have never run more than one GPU in my computers before, so I don't
> >> if there is some special magic that happens when you have two or more
> >> they suddenly get even hotter, but I have to imagine that not to be the
> >> unless you're doing some serious overclocking.
> > Depends on their size. Most GPUs are 2 PCI-e slots high (and occupy two
> > those metal plates on the back) and hence plugging 4 of them in leaves no
> > space inbetween them, which hinders their air intake. Hence the need for
> > watercooling the GPUs in this case.
> That would probably raise your costs significantly, but it would
> greatly increase your odds for success, I'd say. My biggest
> limitations and largest source of problems is that my GPUs eat all 8
> slots. Severely limited my routes for solving problems, and I had to
> cut the plasic off of a portion of a GPU fan shroud to accomodate a
> riser cable. It did end up working, though :)
> >> The ASRock Extreme4 Gen3 does have enough PCIe slots that I could
> >> three GPU's and still have space for a single-slot PCIe device, but I
> >> have a 650W power supply, and have no need for more than one Windows
> >> instance.
> > ... and it has a PLX chip. Right? Or is a PLX chip not a fatal problem?
> I can't speak to an Intel setup, but for a board, I'd suggest the
> Gigabyte GA-990FXA-UD7. I'm testing it right now, and I'm fairly
> confident it will work for what you want to do. The tested and
> working setup that I've built is on the MSI 890FXA-GD70. That board,
> I can 100% guarantee WILL WORK for this purpose (and might require a
> BIOS downgrade, YMMV). I'd recommend the newer Gigabyte board though,
> just because it's newer, I suppose. Probably supports more CPUs or
> >> Secondary pass through works great for gaming, shows the display after
> >> machine boots without any problems, and takes literally no extra effort
> >> setup on your part.
> >> Primary passthrough requires custom ATI patching, and what exists may
> >> work for all cards.
> >> I began looking into Primary passthrough very recently, because I use my
> >> machine for more than just games and ran into a problem. Software like
> >> Photoshop, and 3D Sculpting tools use OpenGL and only work with the
> >> GPU, which means they either don't run or run without GPU acceleration
> >> (slowly).
> > Hmm? I'm not following. Most games also use OpenGL, right? And why would
> > OpenGL not support non-primary cards? I know that OpenCL can run on any
> > number of GPUs, so it'd surprise me if OpenGL was different. Do you have
> > link where I can read more background on this?
> Not sure where to read more on this, but I can share a big caveat:
> GPU-Accelerated video decoding DOES NOT WORK. It crashes stuff.
> Hardcore. Youtube = fail... until you disable hardware decoding. Not
> sure about the OpenGL thing Casey mentions, but I can confirm Direct3D
> works like a charm :)
> >> A lot to take in, but I hope my answers help a bit. If you have more
> >> questions I'll be happy to share what knowledge I can.
> > Certainly, they're of great help. Thanks a lot!
> > Best regards,
> > Peter
> I have to admit I'm excited by the prospect of someone else venturing
> down this path. I just hope that I can save you some repeated bouts
> of banging your head against a wall ;)
> If you'd like, I could send you some photos of my setup, or more
> detailed hardware specs. I had to do a lot of research to build this
> on a reasonable budget... because if it wouldn't have been cheaper
> than multiple systems, I know I wouldn't have done it!
> Andrew Bobulsky