ASRockRack B650D4U Slot 4 PCIe Lanes Not Working
Printed From: ASRock.com
Category: Technical Support
Forum Name: AMD Motherboards
Forum Description: Question about ASRock AMD motherboards
URL: https://forum.asrock.com/forum_posts.asp?TID=27195
Printed Date: 22 Dec 2024 at 12:19am
Software Version: Web Wiz Forums 12.04 - http://www.webwizforums.com
Topic: ASRockRack B650D4U Slot 4 PCIe Lanes Not Working
Posted By: joeschmuckatelli
Subject: ASRockRack B650D4U Slot 4 PCIe Lanes Not Working
Date Posted: 28 Dec 2023 at 2:44am
I have an interesting problem that I have yet to find a solution for and Google has not been of great help, but I will not stop looking even after posting asking for some help. If I find the solution, I will post it for the next unlucky sole.
The problem: I have a SATA add-on card in Slot 7 (x4 slot) however when I bootstrap the computer into ESXi 8.02, I have no network connectivity. Without the card installed all works great. I need to get Slot 7 to play nice.
Why have the add-on card? I need to pass the entire controller through to ESXi with three additional drives attached (two SSD and one HDD) for hobby/development work. I can pass them through using RDM but the VM sees a VMware Drive, not the real physical drive which is what I require.
My system setup: ASRock Rack B650D4U, BIOS 4.01 (provided by ASRock Tech Support), BMC 4.01, 64GB ECC RAM (two 32GB sticks), AMD 7600X CPU, two 4TB NVMe drives installed in the motherboard M.2 slots (one is the boot drive, the other is available for ESXi), four 4TB NVMe drives installed on a NVMe to PCIe add-on card (in slot 6 and bifurcated to x4x4x4x4) which is passed through to ESXi. And all works great until I add the IOCREST SI-PEX40062 into slot 7.
Here is a link to my system build on the TrueNAS website.
https://www.truenas.com/community/threads/my-new-all-nvme-virtualized-truenas-server-story-and-a-guide.114780/" rel="nofollow - https://www.truenas.com/community/threads/my-new-all-nvme-virtualized-truenas-server-story-and-a-guide.114780/
I have never seen such a complicated BIOS setup in my life! I suspect there is something I could disable or enable to make this work. It is odd that my network connectivity goes away, however I can use IPMI over LAN1 (binded to IMPI) without issue.
Examining the B650D4U documentation it clearly states that CPU PCIe lanes 0 - 15 are for Slot 6, PCIe lanes 16 - 19 are for Slot 7, and the last Slot 4 (x1 slot) comes from the PROM21 chipset. I purchased this motherboard based on the technical data in this user guide, as part of my requirement was over 20 PCIe lanes, more is better.
I will be removing the add-on SATA card and placing it into another computer to ensure it is still working but I just removed it from a working computer a few hours ago. However I'm one to verify the hardware didn't fail during the transplant, stranger things have happened.
The second NVMe 4TB drive on the motherboard will likely be removed and placed to the side as a spare, or passed through as an active spare. I'm undecided.
If I missed providing you specific data or some obvious posting that someone else has had, please slap me around, I can handle it.
Cheers,
-Joe
|
Replies:
Posted By: joeschmuckatelli
Date Posted: 29 Dec 2023 at 4:47am
Update:
I pulled the IOCREST SI-PEX40062 "88SE9235 PCIe 2.0 x2 4-port SATA 6 Gb/s Controller" from Slot 7, cleaned the electrical contacts on the edge connector using a pencil eraser to make them all shiny again. Plugged the card into another computer where the card was recognized. I placed the card back into the B650D4U system in Slot 7, still no joy.
I was tired of dealing with this problem for a good part of the day so I gave it a rest and came back to it the following day. I moved the card into Slot 6 (after I returned Slot 6 to x16), We Have Joy! So I moved the card back to Slot 7, We Have Joy. I changed Slot 6 back to x4x4x4x4 and installed my PCIe to M.2 card. Powered on and We Have Joy.
Next, I verified the SATA add-on card was now available in ESXi, I passed through the controller and added it to a VM which all worked as it should.
Lastly, I powered everything off and disconnected the power cord. I waiting ~10 minutes, plugged the power back in, waited for the BMC to fully initialize, turned the system on, ESXi booted up and the SATA controller was passed through again. We have a lot of Joy now. I can close the computer case.
What happened? I will hopefully never need to prove it but I suspect something in the BIOS/PCIe bifurcation caused the issue and changing it from x4x4x4x4 to x16 corrected the issue. I did perform a BIOS and BMC update but I did not reset everything to default/factory, that might be the issue.
|
|