5th Titan Z = 'bF" mobo startup error

Message Topic Search Topic Options Post Reply Create New Topic Printable Version Translate Topic

   3D Rendering rig with 4 Titan Zs, works.  When attempting connection of 5 Titan Zs, receiving "bF" error on mobo prior to BIOS, never proceeds to BIOS.  Can only be remedied by removing and reinstalling CMOS once "bF" error occurs.

Current working setup (7 PCIE Lanes, total 10 GPU cores):1 - GTX Titan X (single core, primary display, direct mobo connection)
2 - unused (covered by Titan X)
3 - GTX Titan Z #1 (dual core GPU, external connection via USB 3.0 PCIE Riser)
4 - GTX Titan Z #2 (dual core GPU, external connection via USB 3.0 PCIE Riser)
5 - GTX Titan Z #3 (dual core GPU, external connection via USB 3.0 PCIE Riser)
6 - GTX Titan Z #4 (dual core GPU, external connection via USB 3.0 PCIE Riser)
7 - GTX 660 Ti (single core GPU, external connection via USB 3.0 PCIE Riser)

Non-working setup resulting in "bF" error (7 PCIE Lanes, total 11 GPU cores):
1 - GTX Titan X (single core, primary display)
2 - unused (covered by Titan X)
3 - GTX Titan Z #1 (dual core GPU, external connection via USB 3.0 PCIE Riser)
4 - GTX Titan Z #2 (dual core GPU, external connection via USB 3.0 PCIE Riser)
5 - GTX Titan Z #3 (dual core GPU, external connection via USB 3.0 PCIE Riser)
6 - GTX Titan Z #4 (dual core GPU, external connection via USB 3.0 PCIE Riser)
7 - GTX Titan Z #5 (dual core GPU, external connection via USB 3.0 PCIE Riser)

Anyone think of what "bF" error represents here, all cards are sufficiently powered (2 external EVGA 1600 W PSUs), all lanes operational, error exists prior to BIOS and Win 7.  Seems like a motherboard limitation with trying to have more than 10 GPU cores?  Could it be a memory issue?  Not sure as it all works with 10 GPU cores, just not 11

Looking online, "bF" seems intended to reference  a chipset program error or a BIOS error...

Thx & Regards!
-GPUnk

Author	Message Topic Search Topic Options Post Reply Create New Topic Printable Version Translate Topic
GPUnk Members Profile Send Private Message Find Members Posts Add to Buddy List Newbie Joined: 03 Jun 2015 Location: CT Status: Offline Points: 19	Post Options Post Reply Quote GPUnk Report Post Thanks(0) Quote Reply Topic: 5th Titan Z = 'bF" mobo startup error Posted: 03 Jun 2015 at 11:53am
	3D Rendering rig with 4 Titan Zs, works. When attempting connection of 5 Titan Zs, receiving "bF" error on mobo prior to BIOS, never proceeds to BIOS. Can only be remedied by removing and reinstalling CMOS once "bF" error occurs. Current working setup (7 PCIE Lanes, total 10 GPU cores): 1 - GTX Titan X (single core, primary display, direct mobo connection) 2 - unused (covered by Titan X) 3 - GTX Titan Z #1 (dual core GPU, external connection via USB 3.0 PCIE Riser) 4 - GTX Titan Z #2 (dual core GPU, external connection via USB 3.0 PCIE Riser) 5 - GTX Titan Z #3 (dual core GPU, external connection via USB 3.0 PCIE Riser) 6 - GTX Titan Z #4 (dual core GPU, external connection via USB 3.0 PCIE Riser) 7 - GTX 660 Ti (single core GPU, external connection via USB 3.0 PCIE Riser) Non-working setup resulting in "bF" error (7 PCIE Lanes, total 11 GPU cores): 1 - GTX Titan X (single core, primary display) 2 - unused (covered by Titan X) 3 - GTX Titan Z #1 (dual core GPU, external connection via USB 3.0 PCIE Riser) 4 - GTX Titan Z #2 (dual core GPU, external connection via USB 3.0 PCIE Riser) 5 - GTX Titan Z #3 (dual core GPU, external connection via USB 3.0 PCIE Riser) 6 - GTX Titan Z #4 (dual core GPU, external connection via USB 3.0 PCIE Riser) 7 - GTX Titan Z #5 (dual core GPU, external connection via USB 3.0 PCIE Riser) Anyone think of what "bF" error represents here, all cards are sufficiently powered (2 external EVGA 1600 W PSUs), all lanes operational, error exists prior to BIOS and Win 7. Seems like a motherboard limitation with trying to have more than 10 GPU cores? Could it be a memory issue? Not sure as it all works with 10 GPU cores, just not 11 Looking online, "bF" seems intended to reference a chipset program error or a BIOS error... Thx & Regards! -GPUnk
	X79 Extreme 11 / Intl Core i7 3930K/ 32gb DDR3 1600 Corsair Vengnce/ Win 7 64bit / Thermaltake 1500 W PSU / GTX Titan X GPU Rendering - 4 Titan Z (external USB 3.0 PCIE riser)/ EVGA 1600 W PSU (2)

Xaltar Members Profile Send Private Message Find Members Posts Add to Buddy List Moderator Group Joined: 16 May 2015 Location: Europe Status: Offline Points: 25073	Post Options Post Reply Quote Xaltar Report Post Thanks(1) Quote Reply Posted: 03 Jun 2015 at 1:06pm
	First of all, that is a beast of a machine As far as I know, dual GPU cards utilize the same number of PCIE lanes as a single GPU card so to my mind the limitation is more likely power related than GPU count though I am no expert in this regard. According to your manual the BF code is "reserved for future AMI codes" basically unspecified so that is of absolutely 0 use. Have you tried removing a different Titan Z other than the one you are trying to install? If there is some sort of GPU count restriction it should work regardless of where the cards are connected. Another suggestion would be to remove the Titan X and try the 660TI as your primary display along with the 5 Titan Zs. The reason I suggest using the 660 as your primary display is that the 660 has a significantly lower TDP than the Titan X so if the problem is power related (power coming from the motherboard via PCIE slots) that my allow the 5 titan Zs to function together and should help determine if the problem is GPU count or power related. The 660 and Titan X are both single GPU cards though the 660 is significantly less power hungry. I suspect you may be dealing with an issue regarding how much power the system can provide via the PCIE slots. Are you using powered PCIE risers? If not that may help eliminate the power part of the equation. I hope this helps. Edited by Xaltar - 03 Jun 2015 at 1:07pm

GPUnk Members Profile Send Private Message Find Members Posts Add to Buddy List Newbie Joined: 03 Jun 2015 Location: CT Status: Offline Points: 19	Post Options Post Reply Quote GPUnk Report Post Thanks(0) Quote Reply Posted: 03 Jun 2015 at 10:59pm
	Hi Xaltar, Thanks for your thoughtful insight. I don't know where to go with it yet though. The PCIE risers are powered through Molex. However, I got excited after reading your message because I saw on my mobo there were 2 molex plugs for Crossfire / SLI, meaning a way to introduce more power to the PCIE lanes. I did plug in the molex on both to the PSU and gave it a try, but still the "bF" error pre-BIOS. Also tried the GTX 660 Ti as primary in place of the X as you recommended, and still got the "bF". With the Z configuration, yeah it could be any combo of 4, but introducing a 5th causes the "bF". I guess, maybe I will ask, issues pre-BIOS, what could that be....power, memory, hardware connection...anything else? I feel I've tried power, and hardware connections, haven't tried anything DRAM, although my hunch tells me it's not that. In the process I learned there is a CMOS reset switch that I can press in place of popping out the actual CMOS circle battery, which is good to know. Also, the molex power to the PCIE lanes probably will help as well....so not a bad learning process thus far...just wish I could get my hands on what would fix the "bF" Instead of saying WTF, I am saying WTbF.... :) Anyway, to you and anyone else, thanks for any other thoughts or suggestions... HOOOOOOOOOOWOOLLLLLLL!!!!! from "the Beast"... :)
	X79 Extreme 11 / Intl Core i7 3930K/ 32gb DDR3 1600 Corsair Vengnce/ Win 7 64bit / Thermaltake 1500 W PSU / GTX Titan X GPU Rendering - 4 Titan Z (external USB 3.0 PCIE riser)/ EVGA 1600 W PSU (2)

Xaltar Members Profile Send Private Message Find Members Posts Add to Buddy List Moderator Group Joined: 16 May 2015 Location: Europe Status: Offline Points: 25073	Post Options Post Reply Quote Xaltar Report Post Thanks(1) Quote Reply Posted: 03 Jun 2015 at 11:27pm
	The only other thing I can think of is trying the 5th Titan Z as your primary in place of the Titan X, that will score you one more GPU, if you still get the bF error then you may well have been right about the GPU limit.

GPUnk Members Profile Send Private Message Find Members Posts Add to Buddy List Newbie Joined: 03 Jun 2015 Location: CT Status: Offline Points: 19	Post Options Post Reply Quote GPUnk Report Post Thanks(0) Quote Reply Posted: 04 Jun 2015 at 12:14pm
	Hi Xaltar, Thank you for your continued thoughts and recommendation. I can confirm that yes, I can run 5 Titan Z's exclusively (1 primary, 4 as secondary level cards for rendering), without any other additional cards (total 10 cores). As such, I am now understanding that this is independent of Titan Z, and a motherboard issue. I actually tried upping the VCCSA voltage in BIOS (I saw other people online doing this to achieve multi GPU stability), in order to possibly ease an 11 core setup on the PC, without any luck. Maybe it would work if I continued allowing more voltage, but I don't know. As backdrop, I use a lead Titan X with its 12GB memory to strictly manipulate and monitor the 3D scenes I create (ie in Daz Studio, Blender, Sketchup, 3DSMax, Octane Render), and let the lighting/rendering be handled by the Z's with all their CUDA. Creating single scene images is no sweat for the cards. But pumping out thousands of them as frames to create an animation is where the cards' real work is, and the more cards the faster it can go, and so I am trying to push it beyond any limit if I can. I actually opened up a support ticket with ASRock, I will see what ASRock themselves say about having a functional Titan X primary with 5 Z's rendering. If you or anyone else has any suggestions or knowledge about BIOS voltage modding to increase stability, I would be thrilled to hear any thoughts. Regards!
	X79 Extreme 11 / Intl Core i7 3930K/ 32gb DDR3 1600 Corsair Vengnce/ Win 7 64bit / Thermaltake 1500 W PSU / GTX Titan X GPU Rendering - 4 Titan Z (external USB 3.0 PCIE riser)/ EVGA 1600 W PSU (2)

Xaltar Members Profile Send Private Message Find Members Posts Add to Buddy List Moderator Group Joined: 16 May 2015 Location: Europe Status: Offline Points: 25073	Post Options Post Reply Quote Xaltar Report Post Thanks(1) Quote Reply Posted: 04 Jun 2015 at 12:49pm
	Great, that means that all your Titan Zs are properly functional and eliminates them from the potential trouble list. I suspect based on your results that its more likely the motherboard does not like 11 GPUs rather than it being a power problem. The bios has specific codes for power, GPU, CPU and memory problems so the fact that the bios is flagging an unspecified error tells me that its likely the bios is attempting something it wasn't designed for or is operating outside of parameters the manufacturer did not foresee. Hopefully Asrock Tech support will chime in here. You have done everything possible to rule out other causes so we are left with the GPU limit seeming most likely. Motherboard, CPU, RAM and all GPUs are functioning correctly, the system is getting plenty of power and there does not appear to be any other problems with the system. I am not overly familiar with multiple GPU setups so I would not be comfortable recommending overvoltage settings for your system, I also doubt it will work based on the tests you have done. You are consistently getting the same error, if it were a case of voltage it is likely the problem would have been more intermittent. I suspect Asrock never anticipated 11 GPUs on that motherboard , not many companies/individuals can afford or are willing to fork out that kind of money for a single system. At least we have made Tech support's job a little easier :)

GPUnk Members Profile Send Private Message Find Members Posts Add to Buddy List Newbie Joined: 03 Jun 2015 Location: CT Status: Offline Points: 19	Post Options Post Reply Quote GPUnk Report Post Thanks(0) Quote Reply Posted: 11 Jun 2015 at 2:48am
	Hi, just an update for any reader of post. I opened support ticket with ASRock, and received response advising I try BIOS P3.30F which has "4G option" in Boot-up section, and to set this parameter to "Enabled". I am assuming enabling a 4G Rom through BIOS would somehow remove limit of GPUs, although the solution was posed to me as more of a "try this out and see if it works" sort of thing. When I attempted, problem was that the suggested BIOS pre-dated my currently used BIOS of P3.30M, which was the first to enable usage of GTX Titan X. When I tried to install the suggested P3.30F BIOS, PC did not boot and I got "b2" error, as this P3.30F BIOS does not support the GTX Titan X. So I had to switch to an older GTX 660 Ti. Once I had access to the BIOS visually, I found my USB mouse, keyboard, and thumb drive in USB 3.0 ports no longer functioned, and I had to switch keyboard, mouse and thumbdrive them to USB 2.0 (weird). But, once I reconnected and had access to the BIOS, I re-installed the P3.30M back again, restoring functionality to the Titan X. What I learned is that P3.30M, which allows the Titan X to function on the board, interestingly does not have the "4G option" in its Boot-up section of the UEFI. I responded to ASRock support suggesting the possible solution of adding the option for 4G rom to the P3.30M BIOS (which allows the GTX Titan X). But, I am wondering if there is something which wouldn't allow the P3.30M BIOS to have the option for 4G, because it is weird that it was not included in that BIOS's UEFI in the first place. Currently awaiting response back! Will update...
	X79 Extreme 11 / Intl Core i7 3930K/ 32gb DDR3 1600 Corsair Vengnce/ Win 7 64bit / Thermaltake 1500 W PSU / GTX Titan X GPU Rendering - 4 Titan Z (external USB 3.0 PCIE riser)/ EVGA 1600 W PSU (2)

Xaltar Members Profile Send Private Message Find Members Posts Add to Buddy List Moderator Group Joined: 16 May 2015 Location: Europe Status: Offline Points: 25073	Post Options Post Reply Quote Xaltar Report Post Thanks(0) Quote Reply Posted: 11 Jun 2015 at 2:15pm
	Did your Titan Zs all pick up and work with the GTX 660 as your primary on the older revision bios or did it still fail to pass post?

GPUnk Members Profile Send Private Message Find Members Posts Add to Buddy List Newbie Joined: 03 Jun 2015 Location: CT Status: Offline Points: 19	Post Options Post Reply Quote GPUnk Report Post Thanks(0) Quote Reply Posted: 25 Jun 2015 at 2:02pm
	Hi Xaltar, thanks for giving me your suggestion. I have confirmed that under older revision BIOS (P3.30F) I can, yes, get PC to boot with 5 Titan Z, with the GTX 660 Ti as the primary display GPU card. There is an option under NorthBridge Configuration "Above 4G Option", by default is "Disabled" and when set to "Enabled" it booted past post and into Windows with 5 Titan Z! Some details during testing: -newest revision BIOS P3.30M, with 4 Titan Z - 660 Ti / Titan X as primary boots successfully -newest revision BIOS P3.30M, with 5 Titan Z - 660 Ti yields beeping error "d4", whereas Titan X yields "bF" -older revision BIOS P3.30F, no 4G enabled, with 4 Titan Z - 660 Ti yields error "bF" -older revision BIOS P3.30F, 4G enabled, with 4 or 5 Titan Z - 660 Ti boots successfully -older revision BIOS P3.30F - Titan X as primary doesn't/can't boot, yields error "b2" Conclusion: GTX Titan X cannot boot as primary GPU with older revision BIOS P3.30F. Newest revision BIOS P3.30M, allows Titan X to boot as primary, however has no option in either Boot Menu or NorthBridge for "Above 4G Option". As such, I cannot try 4G Option with a Titan X in the scenario. I am looking to see if we can get the 4G option added to the newest revision BIOS P3.30M. Xaltar, any thoughts on what I would like to get, or what to convey to ASRock Support (informing them what I've listed above is what I am assuming is the next step...) Again, thanks so much, regards!
	X79 Extreme 11 / Intl Core i7 3930K/ 32gb DDR3 1600 Corsair Vengnce/ Win 7 64bit / Thermaltake 1500 W PSU / GTX Titan X GPU Rendering - 4 Titan Z (external USB 3.0 PCIE riser)/ EVGA 1600 W PSU (2)

Xaltar Members Profile Send Private Message Find Members Posts Add to Buddy List Moderator Group Joined: 16 May 2015 Location: Europe Status: Offline Points: 25073	Post Options Post Reply Quote Xaltar Report Post Thanks(0) Quote Reply Posted: 25 Jun 2015 at 2:39pm
	Happy to be of service I would simply inform them that you would like the "Above 4G Option" option restored to the latest bios revision to enable you to operate the GPU configuration you require. Be sure to formulate your request in a clear and easy to follow way, I have often seen support requests that are so convoluted that I couldn't figure out what the poster actually wanted I think start with asking for the "Above 4G Option" to be restored as you need it to utilize 5 Titan Z GPUs and a Titan X. Only go more in depth if they ask you for further information. What concerns me however is the possibility that the "Above 4g" option may well be automatically enabled in the latest bios and that it is some other change that is preventing the cards from playing nice together. If the option turns out to be enabled by default then I guess you will have to ask them to add support for the Titan X to the bios revision that works. It could get fairly lengthy trying to troubleshoot the latest revision. Edited by Xaltar - 25 Jun 2015 at 2:43pm