3D Rendering rig with 4 Titan Zs, works.  When attempting connection of 5 Titan Zs, receiving "bF" error on mobo prior to BIOS, never proceeds to BIOS.  Can only be remedied by removing and reinstalling CMOS once "bF" error occurs.

1 - GTX Titan X (single core, primary display, direct mobo connection)

2 - unused (covered by Titan X)

3 - GTX Titan Z #1 (dual core GPU, external connection via USB 3.0 PCIE Riser)

Anyone think of what "bF" error represents here, all cards are sufficiently powered (2 external EVGA 1600 W PSUs), all lanes operational, error exists prior to BIOS and Win 7.  Seems like a motherboard limitation with trying to have more than 10 GPU cores?  Could it be a memory issue?  Not sure as it all works with 10 GPU cores, just not 11

Looking online, "bF" seems intended to reference  a chipset program error or a BIOS error...

Thx & Regards!

-GPUnk

As far as I know, dual GPU cards utilize the same number of PCIE lanes as a single GPU card so to my mind the limitation is more likely power related than GPU count though I am no expert in this regard. 

According to your manual the BF code is "reserved for future AMI codes" basically unspecified so that is of absolutely 0 use.

Have you tried removing a different Titan Z other than the one you are trying to install? If there is some sort of GPU count restriction it should work regardless of where the cards are connected. Another suggestion would be to remove the Titan X and try the 660TI as your primary display along with the 5 Titan Zs. The reason I suggest using the 660 as your primary display is that the 660 has a significantly lower TDP than the Titan X so if the problem is power related (power coming from the motherboard via PCIE slots) that my allow the 5 titan Zs to function together and should help determine if the problem is GPU count or power related. The 660 and Titan X are both single GPU cards though the 660 is significantly less power hungry. I suspect you may be dealing with an issue regarding how much power the system can provide via the PCIE slots. Are you using powered PCIE risers? If not that may help eliminate the power part of the equation.

I hope this helps. 

Thanks for your thoughtful insight.  I don't know where to go with it yet though.  The PCIE risers are powered through Molex.  However, I got excited after reading your message because I saw on my mobo there were 2 molex plugs for Crossfire / SLI, meaning a way to introduce more power to the PCIE lanes.  I did plug in the molex on both to the PSU and gave it a try, but still the "bF" error pre-BIOS.

Also tried the GTX 660 Ti as primary in place of the X as you recommended,  and still got the "bF".  With the Z configuration, yeah it could be any combo of 4, but introducing a 5th causes the "bF".

In the process I learned there is a CMOS reset switch that I can press in place of popping out the actual CMOS circle battery, which is good to know.  Also, the molex power to the PCIE lanes probably will help as well....so not a bad learning process thus far...just wish I could get my hands on what would fix the "bF"

Anyway, to you and anyone else, thanks for any other thoughts or suggestions...

HOOOOOOOOOOWOOLLLLLLL!!!!! 

from "the Beast"... :)

Thank you for your continued thoughts and recommendation.  I can confirm that yes, I can run 5 Titan Z's exclusively (1 primary, 4 as secondary level cards for rendering), without any other additional cards (total 10 cores).   

I am not overly familiar with multiple GPU setups so I would not be comfortable recommending overvoltage settings for your system, I also doubt it will work based on the tests you have done. You are consistently getting the same error, if it were a case of voltage it is likely the problem would have been more intermittent. I suspect Asrock never anticipated 11 GPUs on that motherboard , not many companies/individuals can afford or are willing to fork out that kind of money for a single system. 

I opened support ticket with ASRock, and received response advising I try BIOS P3.30F which has "4G option" in Boot-up section, and to set this parameter to "Enabled".  I am assuming enabling a 4G Rom through BIOS would somehow remove limit of GPUs, although the solution was posed to me as more of a "try this out and see if it works" sort of thing.

When I attempted, problem was that the suggested BIOS pre-dated my currently used BIOS of P3.30M, which was the first to enable usage of GTX Titan X.  When I tried to install the suggested P3.30F BIOS, PC did not boot and I got "b2" error, as this P3.30F BIOS does not support the GTX Titan X.  So I had to switch to an older GTX 660 Ti.  Once I had access to the BIOS visually, I found my USB mouse, keyboard, and thumb drive in USB 3.0 ports no longer functioned, and I had to switch keyboard, mouse and thumbdrive them to USB 2.0 (weird).  But, once I reconnected and had access to the BIOS, I re-installed the P3.30M back again, restoring functionality to the Titan X. 

I have confirmed that under older revision BIOS (P3.30F) I can, yes, get PC to boot with 5 Titan Z, with the GTX 660 Ti as the primary display GPU card.  There is an option under NorthBridge Configuration "Above 4G Option", by default is "Disabled" and when set to "Enabled" it booted past post and into Windows with 5 Titan Z!

Some details during testing:

-newest revision BIOS P3.30M, with 5 Titan Z - 660 Ti yields beeping error "d4", whereas Titan X yields "bF"

-older revision BIOS P3.30F, no 4G enabled, with 4 Titan Z - 660 Ti yields error "bF"

Be sure to formulate your request in a clear and easy to follow way, I have often seen support requests that are so convoluted that I couldn't figure out what the poster actually wanted 

I think start with asking for the "Above 4G Option" to be restored as you need it to utilize 5 Titan Z GPUs and a Titan X. Only go more in depth if they ask you for further information.

Just FYI, I had written "Above 4G Option", and it is actually "Above 4G Decoding".

In case you or anyone is interested, here is some literature on the subject:

https://en.wikipedia.org/wiki/PCI_hole

https://en.wikipedia.org/wiki/Memory-mapped_I/O

Regards!

I sent request form for a TitanX-compatible-Above-4G-Decoding option, and received a new BIOS same day, P3.30K.  I tested and it did not work with the Titan X, I received the same "b2" error that I get with all the other non-Titan X compatible BIOSes.  The "b2" error merely comes up because it is the TitanX, which is separate from the "bF", which comes up when >10 GPU are boot up without a 4G Decoding enabled.

When I tested, both the Titan X and 660 Ti booted by themselves as primary with 4G into Windows.

However, when booted > 10 GPU cores, both Titan X and 660 Ti gave "d4" error (PCI resource allocation).

I was able to confirm too, through process of isolation, that all cards are functioning independent of one another.  So under 4G option, if < 10, fine.  If > 10, "d4".

What is interesting is that on the ones with the 4G option that the Titan X could not boot off of, P3.30F and P3.30K, the 660 Ti posted with > 10 GPU under the 4G option.

So, I replied back with all of this (more succinctly, I just said I wanted Titan X to boot >10 GPU cores under 4G like the 660 Ti had booted with >10 cores under 4G, on P3.30F and P3.30K...), I will see what they say.   

I told them I welcome the testing and assistance, they are very responsive 

Received BIOS P3.30O, and it allowed me to boot into Win 7 64 with > 10 GPU attached under the 4G option, while using the Titan X as primary display, which is what I wanted.  I then looked to see what was different in BIOS, and it turned out that in Storage Configuration, Marvell eSata option was disabled in this new BIOS by default, whereas in the other BIOS it was by default enabled.

So, now I am out of BIOS world and into OS world, booted into windows.  All 11 GPU cores are listed in Device manager, BUT 1 core (a Titan Z core, as in 1/2 of a Titan Z) has the Yellow Triangle Error Code 43 thing happening, netting me 10 active GPU cores (1 Titan X, 9 Titan Z).  Doing some research it appears that the BIOS plays a role in allocating IO space for devices, and at root is responsible for # of max GPU.  However it is not exactly clear if I am facing a continuing BIOS issue, or an OS issue of some sort.  I have seen registry hacks attempting to unlock this type of error, with inconsistent results.

I asked the same ASRock support supplying the BIOSes to see what they think.  I asked if there were any mods that might enable this last 11th core.  

Good luck getting that beast of a machine purring like a kitten  

I don't know if they were referring to BIOS or OS as far as the address space goes, but I am going to assume they are speaking of BIOS.

So, I am now asking them if there is a setting that can be modified in BIOS which will allow the user allocate address space to a GPU.  I have no idea if that term 'address space' could mean more power, memory, or what....but I will see how they answer my question.  

You know, there seems to be a lot of people saying things like:

"Windows 7 can only support 4 GPU (then 6, then 8 GPU in other posts)....", or "upgrade to Windows 8, and you will get one more GPU recognized by the OS", or "motherboards only typically support 7-8 GPU."

But, how do people know this, it half feels like vague new territory sometimes.  

Also FYI, not only does device manager show 11 GPU (with one having a yellow triangle), but GPU-Z shows 11 GPU, just that one has an unknown BIOS.  I tried less than 10 GPU will all the cards, and the 1 will always install itself if it was the one previously unrecognized.  But as soon as I go above 10, then a new 11th takes its place, tries to install and cannot.  That feels like a 'resource issue' with Windows 7 address space, but I have no idea what that resource space consists of, or how to stretch that space further.  Anyway, I will update! 

Guess what happened...BSOD.  Oh well....

Looking under resources in windows 8.1 I see there is a check box to allow changing the allocation manually but it is grayed out. The resource tab will also show you conflicts, if you open up the properties of the GPU that is flagged "!" it may give you more info as to where the conflict is coming from. Back in the days of windows 95/98 these problems were a lot more common place and building systems back then often meant spending hours allocating resources to hardware that was trying to share with incompatible devices, especially with purpose built systems like studio/sound setups. I suspect that the checkbox to allow manual allocation may become available in the advent of a conflict. 

Another thing that may be useful to try is uninstalling Xfast RAM. Xfast RAM makes memory in the "memory hole" available to the system. Memory holes in the past were often used as additional resource allocation on certain devices so removing Xfast RAM may give your more resources to play with. I know there used to be an option on old socket 3-7 (486-Pentium) boards to enable/disable the 15-16mb hole for ISA expansion cards but I have not read up on the technicalities in modern boards for quite some time. May be worth a look.

Just so I am getting you, you were imagining me seeing an already existing BIOS modifiable that I would be able to disable right?  Or, were you actually thinking of me downloading the external utility and then tweaking something?

I understand now that the OS offers a user a viewing lens to the BIOS.  I guess a desirable BIOS will let everything load if you 'ask' it to properly, and a 'good' OS will be able to descriptively tell you what the issue specifically is, if there exists one, and give you possible options to rectify it.  It is also more clear to me that there is not ever really a 'GPU-limit', but rather a 'resource limit'. In my case a resource limit may effectively be limiting the # of GPUs, without actually having had an intent to do so, per-say.  It is because of this my search continues in looking to see what that resource is, and if I can negotiate with some part of the BIOS and OS to yield that additional resource.

Kind of imagine it like the Matrix, trying to negotiate with the process to free the GPU core.  Xaltar, you thus far have been both a Morpheus and an Oracle.  Thanks!... 

So someone can help you, can you please post all the  (model numbers ) of your hardware.

Hehe, glad to be of help :)

Xfast RAM is an app  I was suggesting you remove it if you have it installed. It improves performance but also gives access to the "memory hole", removing it may allow additional resources for your GPUs, I doubt it but its worth a shot, you can always install it again if it doesn't help 

"Because the address randomly changed may cause some unexpected symptom or error on the system, our R&D team need to get the whole system including all devices, and then do the further check."

I guess I am now past the 'try this (BIOS)' stage, and if I wanted to get any more insights it would require sending in my system for an eval.  They don't have any ready answers or suggestions, and it appears they think testing would involve random and unexpected phenomenon, which leads me to believe I'm at the resource limit.  I'm not going to send in for testing, maybe I'm past the point of where ASRock's own ready knowledge goes.  But I'll keep my eye out for any new info that comes my way.  By the way, I tried adding the 660Ti as a 12th connected GPU, and I hit the "bF" error again, even with the 4G decoding enabled.  I also tried turning off any peripheral I could find in BIOS (USB 3.0, Audio, Lan), but to no avail.  Anyone interested, my hardware is in my signature.

Xaltar, you actually mentioned music recording in one of your earlier posts, and what is odd is that I never had any issue with removing and reinstalling my Creative Soundblaster X-Fi audio card.  Given that it was going into a PCI lane, I thought it might get muffed up with all the tinkering.  But nope, always works.  Guess my PC likes music production...!  

I have since went on to buy water-blocks for all 6 Titan Z cards, and I have them water-cooled.  That was crazy, a lot of trial and error getting that set up right, but now have it all running.