ASRock.com Homepage
Forum Home Forum Home > Technical Support > AMD Motherboards
  New Posts New Posts RSS Feed - ASRock X370 Pro - Can't POST with 8 GPUs
  FAQ FAQ  Forum Search   Events   Register Register  Login Login

ASRock X370 Pro - Can't POST with 8 GPUs

 Post Reply Post Reply Page  <1234>
Author
Message
MisterJ View Drop Down
Senior Member
Senior Member


Joined: 19 Apr 2017
Status: Offline
Points: 1082
Post Options Post Options   Thanks (0) Thanks(0)   Quote MisterJ Quote  Post ReplyReply Direct Link To This Post Posted: 11 Mar 2018 at 3:50am
Cool, thanks, JohnM!  Enjoy, John.
Fat1 X399 Pro Gaming, TR 1950X, RAID0 3xSamsung SSD 960 EVO, G.SKILL FlareX F4-3200C14Q-32GFX, Win 10 x64 Pro, Enermx Platimax 850, Enermx Liqtech TR4 CPU Cooler, Radeon RX580, BIOS 2.00, 2xHDDs WD
Back to Top
ASRickroll View Drop Down
Newbie
Newbie
Avatar

Joined: 28 Feb 2018
Location: Japan
Status: Offline
Points: 10
Post Options Post Options   Thanks (0) Thanks(0)   Quote ASRickroll Quote  Post ReplyReply Direct Link To This Post Posted: 11 Mar 2018 at 3:37pm
Thanks everyone Smile

The original BIOS that shipped with my board (v3.10) did not have "Above 4G Decoding" option.
However I updated the BIOS to v4.40 prior and the option now shows up.
What I did try was to boot with 7 GPUs, enter BIOS and enabled the option, save, restart, turn off, insert the eight card.
It didn't help though, but I haven't tried it with CSM disabled and UEFI boot disk (is this a necessary step).

I am too tired now and actually resorted to putting the 8th GPU in another PC. I will give it a try again later next week.

Appreciate the help guys

I hope ASRock engineers hear us


Originally posted by benb benb wrote:

Did you get a custom bios that enables "Above 4G Decoding" from Asrock? 

They can build it for you, wish they would just let it in all their bios. 
They gave me a custom bios a while ago that enabled the flag, but it is 2.12 ish . I am going to ask them for an updated bios because the newer agesa is suppose to address some of the linux/ryzen issues for early series chips. 

 
"Above 4G Decoding" works as  long as CSM is disabled or if enabled only with UEFI OpRoms(There are three, PXE,Storage and Something else) 


You might want to disable CSM first , ie remove some cards boot , disable CSM and only allow UEFI to load. 


Hopefully if we bother Asrock enough they will just roll the "Above 4G Decoding" out to their bios with a comment  "For mining/advance users", something along those lines. 

Biostar and MSI have rolled out the option to their bioses and I think gigabyte also. Only ASUS and Asrock for some reason aren't rolling it out. 

Example MSI rolled "Above 4G Decoding" out to all their x370 motherboards. http://www.nag.co.za/2017/07/14/msi-revamps-their-bios-to-support-crypto-mining-farms-better/



Do you have contact details of ASRock BIOS engineers team? I want to bother them as well Smile


Edited by ASRickroll - 11 Mar 2018 at 3:38pm
Back to Top
MisterJ View Drop Down
Senior Member
Senior Member


Joined: 19 Apr 2017
Status: Offline
Points: 1082
Post Options Post Options   Thanks (0) Thanks(0)   Quote MisterJ Quote  Post ReplyReply Direct Link To This Post Posted: 12 Mar 2018 at 12:00am
ASRickroll, seems like I read that UEFI is required.  If so, Disable CSM then you must have W10 installed in UEFI mode which requires a GPT Style system disk.  You may need to reinstall W10.  Pleas tell us where the  "Above 4G Decoding" option is in your BIOS/UEFI.  Thanks and enjoy, John.
Fat1 X399 Pro Gaming, TR 1950X, RAID0 3xSamsung SSD 960 EVO, G.SKILL FlareX F4-3200C14Q-32GFX, Win 10 x64 Pro, Enermx Platimax 850, Enermx Liqtech TR4 CPU Cooler, Radeon RX580, BIOS 2.00, 2xHDDs WD
Back to Top
benb View Drop Down
Newbie
Newbie


Joined: 29 Jun 2017
Status: Offline
Points: 11
Post Options Post Options   Thanks (1) Thanks(1)   Quote benb Quote  Post ReplyReply Direct Link To This Post Posted: 12 Mar 2018 at 6:02am
First lets see if the board can initialize properly with all the video cards. 
0) Clear your CMOS , want to clear out anything the bios has learned about how to initialize the PCIE bus. 

1) Disconnect your boot drive.
1a) We just want to make sure the board can initialize the PCI-E bus properly with all those cards, which has nothing to do with the OS. 
2)Boot to the bios configuration , go to Boot-> the CSM -> Change each one to UEFI only ,(Launch PXE, Launch Storage, Launch Video)
3) Change the Boot Option 1 to be UEFI: Built-in EFI Shell
4) Enable "Above 4G Decoding"
5) Save the configuration and reboot. 

The machine might reset a few times as it tries to learn how to initialize the PCIE bus. 

When the machine stabilizes to a constant Dr Debug code, look at the code. Match the code up with one from here https://www.asrock.com/support/faq.asp?id=334 . 

A0-A7 just means it is waiting for sata disk with UEFI load. 

If you get 92-99 that is PCI-E error, so try starting with say 3 video cards first and seeing if the PCI-E bus initializes correctly.  
Back to Top
JohnM View Drop Down
Groupie
Groupie
Avatar

Joined: 20 Feb 2018
Location: UK
Status: Offline
Points: 267
Post Options Post Options   Thanks (0) Thanks(0)   Quote JohnM Quote  Post ReplyReply Direct Link To This Post Posted: 12 Mar 2018 at 11:28am
If you do need to change to UEFI boot you don't need to reinstall Windows 10. You can run the mbr2gpt.exe command that's included in the OS and it converts an existing MBR partitioned drive to GPT. I used it myself and it works. Details are here:  https://www.intel.co.uk/content/www/uk/en/support/articles/000024558/memory-and-storage/intel-optane-memory.html
ASRock Fatal1ty AB350 ITX P4.90, AMD Ryzen 5 2400G, 2x8GB Corsair CMK16GX4M2A2666C16, 250GB Samsung 960EVO, 500GB Samsung 850EVO, 4TB WD Blue, Windows 10 Pro 64, Corsair SF450, Cooler Master Elite 110
Back to Top
ASRickroll View Drop Down
Newbie
Newbie
Avatar

Joined: 28 Feb 2018
Location: Japan
Status: Offline
Points: 10
Post Options Post Options   Thanks (0) Thanks(0)   Quote ASRickroll Quote  Post ReplyReply Direct Link To This Post Posted: 12 Mar 2018 at 11:52am
Originally posted by MisterJ MisterJ wrote:

ASRickroll, seems like I read that UEFI is required.  If so, Disable CSM then you must have W10 installed in UEFI mode which requires a GPT Style system disk.  You may need to reinstall W10.  Pleas tell us where the  "Above 4G Decoding" option is in your BIOS/UEFI.  Thanks and enjoy, John.

I just checked again. The name of the option is actually "Above 4G MMIO Enable" which is disabled by default. I wonder if this is the same thing?


Originally posted by benb benb wrote:

First lets see if the board can initialize properly with all the video cards. 
0) Clear your CMOS , want to clear out anything the bios has learned about how to initialize the PCIE bus. 

1) Disconnect your boot drive.
1a) We just want to make sure the board can initialize the PCI-E bus properly with all those cards, which has nothing to do with the OS. 
2)Boot to the bios configuration , go to Boot-> the CSM -> Change each one to UEFI only ,(Launch PXE, Launch Storage, Launch Video)
3) Change the Boot Option 1 to be UEFI: Built-in EFI Shell
4) Enable "Above 4G Decoding"
5) Save the configuration and reboot. 

The machine might reset a few times as it tries to learn how to initialize the PCIE bus. 

When the machine stabilizes to a constant Dr Debug code, look at the code. Match the code up with one from here https://www.asrock.com/support/faq.asp?id=334 . 

A0-A7 just means it is waiting for sata disk with UEFI load. 

If you get 92-99 that is PCI-E error, so try starting with say 3 video cards first and seeing if the PCI-E bus initializes correctly.  

Thanks benb, I will try that.
The new option that shows in recent BIOS versions is "Above 4G MMIO Enable" (disabled by default) - is this the same thing as "Above 4G Decoding"? Smile
Back to Top
benb View Drop Down
Newbie
Newbie


Joined: 29 Jun 2017
Status: Offline
Points: 11
Post Options Post Options   Thanks (0) Thanks(0)   Quote benb Quote  Post ReplyReply Direct Link To This Post Posted: 13 Mar 2018 at 4:50am

Above 4G MMIO is the same as above Above 4G decoding. 


The 4G MMIO Limit , needs to be less than 42 bits for NVIDIA cards. 40 Bits the default should be fine. 

Only thing I see that might be an issue is usually if you have the MMIO Limit bits you also have a another option for the Base Address. 

Basically this sets up a range of addresses the gpu cards are mapped to. 
Here is an example of what I am talking about on a supermicro motherboard. https://www.servethehome.com/nvidia-smi-issues-get-nvidia-cuda-working-grid-tesla-gpus/


Anyways I had to roll back to my custom 2.10 version because I couldn't get it to boot with the Xeon Phi card , I have a feeling it is MMIO Limit bits need to be correct or there is some issue with the bios. 
I get a bios error 07 , which is memory error. Basically the Xeon Phi gets memory mapped, and I am pretty sure that this memory mapping is incorrect. 


BIOS might be fine for Nvidia gpus with above 4g decoding, Xeon Phi is very picky. Bad thing is the Xeon Phi keeps the machine from posting at all. 
Back to Top
MisterJ View Drop Down
Senior Member
Senior Member


Joined: 19 Apr 2017
Status: Offline
Points: 1082
Post Options Post Options   Thanks (0) Thanks(0)   Quote MisterJ Quote  Post ReplyReply Direct Link To This Post Posted: 13 Mar 2018 at 5:15am
Thanks, ASRickroll.  I suspect my board does not have the option but would like to ask under what major section is "Above 4G MMIO Enable".  Something like Advanced-AMD PBS.  I am still running UEFI 2.00 since ASRock has not released anything for X399 since 11-28-2017.  Thanks and enjoy, John.
Fat1 X399 Pro Gaming, TR 1950X, RAID0 3xSamsung SSD 960 EVO, G.SKILL FlareX F4-3200C14Q-32GFX, Win 10 x64 Pro, Enermx Platimax 850, Enermx Liqtech TR4 CPU Cooler, Radeon RX580, BIOS 2.00, 2xHDDs WD
Back to Top
pbfarmer View Drop Down
Newbie
Newbie


Joined: 09 Mar 2018
Status: Offline
Points: 8
Post Options Post Options   Thanks (0) Thanks(0)   Quote pbfarmer Quote  Post ReplyReply Direct Link To This Post Posted: 13 Mar 2018 at 5:36pm
Originally posted by benb benb wrote:


Above 4G MMIO is the same as above Above 4G decoding. 


The 4G MMIO Limit , needs to be less than 42 bits for NVIDIA cards. 40 Bits the default should be fine. 

Only thing I see that might be an issue is usually if you have the MMIO Limit bits you also have a another option for the Base Address. 

Basically this sets up a range of addresses the gpu cards are mapped to. 
Here is an example of what I am talking about on a supermicro motherboard. https://www.servethehome.com/nvidia-smi-issues-get-nvidia-cuda-working-grid-tesla-gpus/


Anyways I had to roll back to my custom 2.10 version because I couldn't get it to boot with the Xeon Phi card , I have a feeling it is MMIO Limit bits need to be correct or there is some issue with the bios. 
I get a bios error 07 , which is memory error. Basically the Xeon Phi gets memory mapped, and I am pretty sure that this memory mapping is incorrect. 


BIOS might be fine for Nvidia gpus with above 4g decoding, Xeon Phi is very picky. Bad thing is the Xeon Phi keeps the machine from posting at all. 

Sooo....  After days of tinkering, I've hit a dead end.  Figured out my CSM setting was getting auto-reset because my GPUs were using custom roms with invalid UEFI signatures.  Fixed all the cards, updated my Win10 install to GPT/UEFI, and finally got the CSM setting to stick, still no 8th card.  

Then discovered (between this thread and a support email) that the latest UEFI rom for my motherboard (Fatal1ty X370 Gaming K4) contains the 4G MMIO setting, so I installed the latest rom and played around w/ install order of cards for a while.  Actually got the board to POST w/ 8 cards plugged in, but the system never actually presented the 8th card - seems resources were never assigned.  I think this minor win was more due to the order I was plugging in the cards than the 4G setting, but it doesn't really matter, since the 8th card was still non-functional.

Digging into the Device Manager and System Info in Windows, as well as the UEFI shell, I've confirmed that the UEFI is never assigning resources above 4G, regardless of the Above 4G MMIO setting.  Other settings including MMIO bits, CSM and PCIe lane speed have no effect on this either.

It appears @benb is correct - there needs to be an additional setting in the UEFI for this Above 4G setting to be useful.  Either an explicit control to define a >4G base address, or a control to restrict the <4G available space, forcing assignments above 4G.  There's mention in other forums of a TOLUD (top of lower usable dram) setting on some ASRock Intel boards, which sounds like it performs exactly this function.  I guess I need to go back to support and ask about this additional setting now.

@benb - can you confirm (screenshot maybe) that your GPU(s) are indeed using >4G addresses w/ your custom rom?  Also, what board are you using, and could you possibly post your rom somewhere?  Even if I can't use it, i'd like to see if I can pick out the change they made using a hex editor.  Thanks
Back to Top
MisterJ View Drop Down
Senior Member
Senior Member


Joined: 19 Apr 2017
Status: Offline
Points: 1082
Post Options Post Options   Thanks (0) Thanks(0)   Quote MisterJ Quote  Post ReplyReply Direct Link To This Post Posted: 13 Mar 2018 at 11:09pm
I'm wondering if any of you have run GPU-Z ( https://www.techpowerup.com/gpuz/ ) to see if it tells you anything of interest.  Worth a try?  Enjoy, John.
Fat1 X399 Pro Gaming, TR 1950X, RAID0 3xSamsung SSD 960 EVO, G.SKILL FlareX F4-3200C14Q-32GFX, Win 10 x64 Pro, Enermx Platimax 850, Enermx Liqtech TR4 CPU Cooler, Radeon RX580, BIOS 2.00, 2xHDDs WD
Back to Top
 Post Reply Post Reply Page  <1234>
  Share Topic   

Forum Jump Forum Permissions View Drop Down

Forum Software by Web Wiz Forums® version 12.04
Copyright ©2001-2021 Web Wiz Ltd.

This page was generated in 0.078 seconds.