ASRock.com Homepage
Forum Home Forum Home > Technical Support > AMD Motherboards
  New Posts New Posts RSS Feed - Taichi x399 iommu groups
  FAQ FAQ  Forum Search Search  Events   Register Register  Login Login

Taichi x399 iommu groups
 Post Reply Post Reply
Author
Message
  Topic Search Topic Search  Topic Options Topic Options
Våge View Drop Down
Newbie
Newbie


Joined: 07 Nov 2017
Status: Offline
Points: 5 		Post Options Post Options   Thanks (0) Thanks(0)   Quote Våge Quote  Post ReplyReply Direct Link To This Post Topic: Taichi x399 iommu groups
    Posted: 07 Nov 2017 at 1:45pm
[URL=][/URL]Hi
Can someone list the iommu for the taichi x399?
Are both the network lans in the same group?
Interested in linux with passthrough

Edited by Våge - 07 Nov 2017 at 1:48pm
Back to Top
Akiko View Drop Down
Newbie
Newbie
Avatar

Joined: 27 Sep 2017
Status: Offline
Points: 8 		Post Options Post Options   Thanks (0) Thanks(0)   Quote Akiko Quote  Post ReplyReply Direct Link To This Post Posted: 09 Nov 2017 at 8:22pm
You won't like this.

For the the network stuff it looks like this (even with the ACS UEFI option, version 1.70):
/sys/kernel/iommu_groups/11/devices/0000:06:00.0 <- lower port
/sys/kernel/iommu_groups/11/devices/0000:02:05.0
/sys/kernel/iommu_groups/11/devices/0000:05:00.0 <- wifi
/sys/kernel/iommu_groups/11/devices/0000:01:00.1
/sys/kernel/iommu_groups/11/devices/0000:04:00.0 <- upper port
/sys/kernel/iommu_groups/11/devices/0000:02:04.0
/sys/kernel/iommu_groups/11/devices/0000:02:07.0
/sys/kernel/iommu_groups/11/devices/0000:02:00.0
/sys/kernel/iommu_groups/11/devices/0000:01:00.2
/sys/kernel/iommu_groups/11/devices/0000:01:00.0
/sys/kernel/iommu_groups/11/devices/0000:02:06.0


Also PCI-passthrough for graphic cards is completely broken in KVM (vfio), Xen and even VirtualBox. All cards suffer from getting stuck in D3cold powersave mode. You also have to set the Promotory PCIe connection (the one between the chipset and the CPU) to PCIe gen2 or you will suffer from data link level issues (PCIe AER) spaming the kernel, onboard network not working properly and some very serious performance issues. The whole PCIe PME (power management) looks like a complete fuckup (at least on Linux). And this is true for all UEFI versions up to 1.70 (didn't had the time to test 1.80 yet). It is also true for all Kernels starting from 4.10 to 4.14.rc8. I'm using the last one, because that one works best with all iommu/kvm features and I patched it to have working NPT (an old AMD KVM issue) and TR4 temperatures.


Edited by Akiko - 09 Nov 2017 at 8:39pm
loading failed @ 0xD3ADB33F
Back to Top
Våge View Drop Down
Newbie
Newbie


Joined: 07 Nov 2017
Status: Offline
Points: 5 		Post Options Post Options   Thanks (0) Thanks(0)   Quote Våge Quote  Post ReplyReply Direct Link To This Post Posted: 09 Nov 2017 at 9:13pm
Thx for the info!
Back to Top
Akiko View Drop Down
Newbie
Newbie
Avatar

Joined: 27 Sep 2017
Status: Offline
Points: 8 		Post Options Post Options   Thanks (0) Thanks(0)   Quote Akiko Quote  Post ReplyReply Direct Link To This Post Posted: 10 Nov 2017 at 3:15am
The new 1.80 UEFI fixes some of the PCIe PME issues. The Promotory Link now works in gen3 mode and the PCIe AER spaming is gone and with that the network and performance issues. But PCI-passthrough is still broken and it looks like that.

[  269.943260] vfio-pci 0000:08:00.0: enabling device (0000 -> 0003)
[  269.943393] vfio_ecap_init: 0000:08:00.0 hiding ecap 0x19@0x270
[  271.333626] vfio_bar_restore: 0000:08:00.1 reset recovery - restoring bars
[  272.094766] vfio-pci 0000:08:00.1: timed out waiting for pending transaction; performing function level reset anyway
[  273.495402] vfio-pci 0000:08:00.1: not ready 1123ms after FLR; waiting
[  274.565893] vfio-pci 0000:08:00.1: not ready 2147ms after FLR; waiting
[  276.646855] vfio-pci 0000:08:00.1: not ready 4195ms after FLR; waiting
[  280.808743] vfio-pci 0000:08:00.1: not ready 8291ms after FLR; waiting
[  289.132522] vfio-pci 0000:08:00.1: not ready 16483ms after FLR; waiting
[  305.780097] vfio-pci 0000:08:00.1: not ready 32867ms after FLR; waiting
[  339.075263] vfio-pci 0000:08:00.1: not ready 65635ms after FLR; giving up
[  339.080020] vfio_bar_restore: 0000:08:00.0 reset recovery - restoring bars
[  339.155716] br0: port 3(tap1) entered disabled state
[  339.995665] vfio-pci 0000:08:00.1: timed out waiting for pending transaction; performing function level reset anyway
[  341.396299] vfio-pci 0000:08:00.1: not ready 1123ms after FLR; waiting
[  342.516819] vfio-pci 0000:08:00.1: not ready 2147ms after FLR; waiting
[  344.597759] vfio-pci 0000:08:00.1: not ready 4195ms after FLR; waiting
[  348.759655] vfio-pci 0000:08:00.1: not ready 8291ms after FLR; waiting
[  357.003420] vfio-pci 0000:08:00.1: not ready 16483ms after FLR; waiting
[  373.650985] vfio-pci 0000:08:00.1: not ready 32867ms after FLR; waiting
[  406.946136] vfio-pci 0000:08:00.1: not ready 65635ms after FLR; giving up


After vfio failed to bring up the graphics card, the whole PCI device is in an undefined state.

# lspci -nnvvs 08:00.0
08:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750/8740 / R7 250E] [1002:683f] (rev ff) (prog-if ff)
        !!! Unknown header type 7f
        Kernel driver in use: vfio-pci
        Kernel modules: radeon, amdgpu

# lspci -nnvvs 08:00.1
08:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0] (rev ff) (prog-if ff)
        !!! Unknown header type 7f
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel
loading failed @ 0xD3ADB33F
Back to Top
Våge View Drop Down
Newbie
Newbie


Joined: 07 Nov 2017
Status: Offline
Points: 5 		Post Options Post Options   Thanks (0) Thanks(0)   Quote Våge Quote  Post ReplyReply Direct Link To This Post Posted: 10 Nov 2017 at 3:24am
:(
Thx!
Then i will go for a fatal1ty x399 that seems doing better as per Level1 Linux channel
Back to Top
Akiko View Drop Down
Newbie
Newbie
Avatar

Joined: 27 Sep 2017
Status: Offline
Points: 8 		Post Options Post Options   Thanks (1) Thanks(1)   Quote Akiko Quote  Post ReplyReply Direct Link To This Post Posted: 10 Nov 2017 at 1:10pm
I'm looking into this issues for quite a while now. All X399 boards are affected by this. The only way to get it working is by using a Vega card, where the reset function (FLR) is broken. These cards actually do not go into D3cold powersave mode because of their faulty BIOS code.
loading failed @ 0xD3ADB33F
Back to Top
LevitateMe View Drop Down
Newbie
Newbie


Joined: 10 Nov 2017
Status: Offline
Points: 1 		Post Options Post Options   Thanks (0) Thanks(0)   Quote LevitateMe Quote  Post ReplyReply Direct Link To This Post Posted: 10 Nov 2017 at 2:12pm
Originally posted by Akiko Akiko wrote:


Also PCI-passthrough for graphic cards is completely broken in KVM (vfio), Xen and even VirtualBox.


So far, the only success on threadripper passthrough that seems to be reliable is ESXi but only on motherboards with 6 SATA ports.  For some reason, the 8 port SATA port mobos can't pass dedicated storage devices through.... which means it doesn't work on Taichi. :(
Back to Top
Akiko View Drop Down
Newbie
Newbie
Avatar

Joined: 27 Sep 2017
Status: Offline
Points: 8 		Post Options Post Options   Thanks (0) Thanks(0)   Quote Akiko Quote  Post ReplyReply Direct Link To This Post Posted: 10 Nov 2017 at 6:37pm
This is easy to explain. Look up into my post with the iommu group 11, there the device 01:00.1 is actually the SATA controller with the real SATA ports. So it shares the same iommu group with all the network cards. If you want to use this, you have to pass all the devices (looks like the whole southbridge) of the group into the VM or find a way to split it up a little bit more. Actually the ACS option in the UEFI should exactly do this, but no, the option doesn't seem work right.
loading failed @ 0xD3ADB33F
Back to Top
Akiko View Drop Down
Newbie
Newbie
Avatar

Joined: 27 Sep 2017
Status: Offline
Points: 8 		Post Options Post Options   Thanks (1) Thanks(1)   Quote Akiko Quote  Post ReplyReply Direct Link To This Post Posted: 19 Nov 2017 at 2:08pm
Originally posted by Akiko Akiko wrote:

The new 1.80 UEFI fixes some of the PCIe PME issues. The Promotory Link now works in gen3 mode and the PCIe AER spaming is gone and with that the network and performance issues.


This is actually not true. It looks like it got worse, now this issues even appear in gen2 mode. But the problem completely disappears by turning off the whole ASPM feature of the PCIe subsystem (kernel parameter: pcie_aspm=off). I'm not sure If performance is affected, but USB transactions seem a bit slow.

That turning off ASPM helps is quite interesting, because the ASPM code of current Linux is known for being in a really bad shape. There will be a big update coming with the 4.15 kernel, fixing a lot of powersave, PME and ASPM stuff.
loading failed @ 0xD3ADB33F
Back to Top
 Post Reply Post Reply
  Share Topic   

Forum Jump Forum Permissions View Drop Down

Forum Software by Web Wiz Forums® version 12.04
Copyright ©2001-2021 Web Wiz Ltd.

This page was generated in 6.609 seconds.