AMD主板的几个疑问

Brunt1104 发表于 2023-11-16 16:00

主板是华擎B550钢铁传奇，NAS系统是TrueNAS Scale，插了2条纯ECC内存已经连续开机了两年半很稳定，现在想用NAS开虚拟机当软路由，买了双口光纤网卡插在直连CPU的PCIE×16上，4口千兆网卡插在芯片组出的PCIE×1上，×16接口的网卡可以开启SR-IOV并且直通给虚拟机，×1的4口千兆网卡不管怎么配置都不能直通，网卡换成VirtIO半虚拟化之后LAN口桥接绑定几个接口只有第1个接口能通，不知道是不是固件问题，已经在BIOS开启了所有关于SR-IOV IOMMU ACS的选项都不行，cmdline中也没有ACS参数，查了下好像是B550芯片组不支持ACS拆分，只能换X570主板或者换平台，也有可能是TrueNAS的内核不支持ACS拆分[生病]

在海鲜市场找到一张X570 Unity价格很棒，但微星AMD主板好像不支持ECC功能，华硕和华擎是明确支持的，找到个帖子说BIOS设置中可以开启ECC功能，但我问了卖家没有 OC-Misc-ECC DRAM ENABLE这个选项，不知道是不是要支持ECC的CPU和内存才会显示ECC功能，还是说微星中高端X570主板不支持ECC内存[流汗]

dcl2009 发表于 2023-11-16 16:10

AMD的只有直连CPU的那些PCIE可以直通，其他的均无法直通，SR-IOV也是一样

大头吃小头 发表于 2023-11-16 16:12

dcl2009 发表于 2023-11-16 16:10
AMD的只有直连CPU的那些PCIE可以直通，其他的均无法直通，SR-IOV也是一样

我的x550t2就能直通啊..... 挂在南桥x4上的

Brunt1104 发表于 2023-11-16 16:13

大头吃小头发表于 2023-11-16 16:12
我的x550t2就能直通啊..... 挂在南桥x4上的

老哥你用的是什么系统，是X570主板吗

大头吃小头 发表于 2023-11-16 16:13

pve是有acs补丁的，开了之后iommu都拆的非常细了（我甚至主板BIOS里似乎都没开iommu 支持）

大头吃小头 发表于 2023-11-16 16:14

Brunt1104 发表于 2023-11-16 16:13
老哥你用的是什么系统，是X570主板吗

雕牌b550

不过雕牌没有sriov相关选项..... 开不了sriov

danety 发表于 2023-11-16 16:15

本帖最后由 danety 于 2023-11-16 16:17 编辑

#
# These variables are device properties.For people who are very
# curious about this, you can download the Windows Driver Kit headers and
# look for pciprop.h.All of these are contained in that file.
#
$devpkey_PciDevice_DeviceType = "{3AB22E31-8264-4b4e-9AF5-A8D2D8E33E62}1"
$devpkey_PciDevice_RequiresReservedMemoryRegion = "{3AB22E31-8264-4b4e-9AF5-A8D2D8E33E62}34"
$devpkey_PciDevice_AcsCompatibleUpHierarchy = "{3AB22E31-8264-4b4e-9AF5-A8D2D8E33E62}31"

$devprop_PciDevice_DeviceType_PciConventional                      = 0
$devprop_PciDevice_DeviceType_PciX                               = 1
$devprop_PciDevice_DeviceType_PciExpressEndpoint                   = 2
$devprop_PciDevice_DeviceType_PciExpressLegacyEndpoint             = 3
$devprop_PciDevice_DeviceType_PciExpressRootComplexIntegratedEndpoint= 4
$devprop_PciDevice_DeviceType_PciExpressTreatedAsPci             = 5
$devprop_PciDevice_BridgeType_PciConventional                      = 6
$devprop_PciDevice_BridgeType_PciX                               = 7
$devprop_PciDevice_BridgeType_PciExpressRootPort                   = 8
$devprop_PciDevice_BridgeType_PciExpressUpstreamSwitchPort       = 9
$devprop_PciDevice_BridgeType_PciExpressDownstreamSwitchPort       =10
$devprop_PciDevice_BridgeType_PciExpressToPciXBridge             =11
$devprop_PciDevice_BridgeType_PciXToExpressBridge                =12
$devprop_PciDevice_BridgeType_PciExpressTreatedAsPci             =13
$devprop_PciDevice_BridgeType_PciExpressEventCollector             =14

$devprop_PciDevice_AcsCompatibleUpHierarchy_NotSupported          = 0
$devprop_PciDevice_AcsCompatibleUpHierarchy_SingleFunctionSupported= 1
$devprop_PciDevice_AcsCompatibleUpHierarchy_NoP2PSupported       = 2
$devprop_PciDevice_AcsCompatibleUpHierarchy_Supported             = 3

write-host "Generating a list of PCI Express endpoint devices"
$pnpdevs = Get-PnpDevice -PresentOnly
$pcidevs = $pnpdevs | Where-Object {$_.InstanceId -like "PCI*"}

foreach ($pcidev in $pcidevs) {
Write-Host ""
Write-Host ""
Write-Host -ForegroundColor White -BackgroundColor Black $pcidev.FriendlyName

$rmrr =($pcidev | Get-PnpDeviceProperty $devpkey_PciDevice_RequiresReservedMemoryRegion).Data
if ($rmrr -ne 0) {
   write-host -ForegroundColor Red -BackgroundColor Black "BIOS requires that this device remain attached to BIOS-owned memory.Not assignable."
   continue
}

$acsUp =($pcidev | Get-PnpDeviceProperty $devpkey_PciDevice_AcsCompatibleUpHierarchy).Data
if ($acsUp -eq $devprop_PciDevice_AcsCompatibleUpHierarchy_NotSupported) {
   write-host -ForegroundColor Red -BackgroundColor Black "Traffic from this device may be redirected to other devices in the system.Not assignable."
   continue
}

$devtype = ($pcidev | Get-PnpDeviceProperty $devpkey_PciDevice_DeviceType).Data
if ($devtype -eq $devprop_PciDevice_DeviceType_PciExpressEndpoint) {
   Write-Host "Express Endpoint -- more secure."
} else {
   if ($devtype -eq $devprop_PciDevice_DeviceType_PciExpressRootComplexIntegratedEndpoint) {
         Write-Host "Embedded Endpoint -- less secure."
   } else {
         if ($devtype -eq $devprop_PciDevice_DeviceType_PciExpressTreatedAsPci) {
            Write-Host -ForegroundColor Red -BackgroundColor Black "BIOS kept control of PCI Express for this device.Not assignable."
         } else {
            Write-Host -ForegroundColor Red -BackgroundColor Black "Old-style PCI device, switch port, etc.Not assignable."
         }
         continue
   }
}

$locationpath = ($pcidev | get-pnpdeviceproperty DEVPKEY_Device_LocationPaths).data

#
# Now do a check for the interrupts that the device uses.Line-based interrupts
# aren't assignable.
#
$doubleslashDevId = "*" + $pcidev.PNPDeviceID.Replace("\","\\") + "*"
$irqAssignments = gwmi -query "select * from Win32_PnPAllocatedResource" | Where-Object {$_.__RELPATH -like "*Win32_IRQResource*"} | Where-Object {$_.Dependent -like $doubleslashDevId}

#$irqAssignments | Format-Table -Property __RELPATH

if ($irqAssignments.length -eq 0) {
   Write-Host -ForegroundColor Green -BackgroundColor Black " And it has no interrupts at all -- assignment can work."
} else {
   #
   # Find the message-signaled interrupts.They are reported with a really big number in
   # decimal, one which always happens to start with "42949...".
   #
   $msiAssignments = $irqAssignments | Where-Object {$_.Antecedent -like "*IRQNumber=42949*"}

   #$msiAssignments | Format-Table -Property __RELPATH

   if ($msiAssignments.length -eq 0) {
         Write-Host -ForegroundColor Red -BackgroundColor Black "All of the interrupts are line-based, no assignment can work."
         continue
   } else {
         Write-Host -ForegroundColor Green -BackgroundColor Black " And its interrupts are message-based, assignment can work."
   }
}

#
# Print out the location path, as that's the way to refer to this device that won't
# change even if you add or remove devices from the machine or change the way that
# the BIOS is configured.
#
$locationpath
}

#
# Now look at the host as a whole.Asking whether the host supports SR-IOV
# is mostly equivalent to asking whether it supports Discrete Device
# Assignment.
#
if ((Get-VMHost).IovSupport -eq $false) {
Write-Host ""
write-host "Unfortunately, this machine doesn't support using them in a VM."
Write-Host ""
(Get-VMHost).IovSupportReasons
}

上面文件保存为ps文件，在powershell里运行一下，每一个pcie设备支不支持直通，以及不支持的原因都会显示出来
直通既需要主板的支持也需要pcie设备的支持

dcl2009 发表于 2023-11-16 16:16

大头吃小头发表于 2023-11-16 16:12
我的x550t2就能直通啊..... 挂在南桥x4上的

你用的是PVE吧？ESXI确实只能直通CPU的那20条PCIE

Brunt1104 发表于 2023-11-16 16:17

danety 发表于 2023-11-16 16:15
#
# These variables are device properties.For people who are very
# curious about this, you can...

NAS系统是TrueNAS Scale，有Linux的脚本吗

大头吃小头 发表于 2023-11-16 16:18

dcl2009 发表于 2023-11-16 16:16
你用的是PVE吧？ESXI确实只能直通CPU的那20条PCIE

是的，pve7.4

dcl2009 发表于 2023-11-16 16:20

大头吃小头发表于 2023-11-16 16:18
是的，pve7.4

PVE确实可以直通，连核显都能直通，可惜有的设备不是很稳定，我用过一段时间，后来转回ESXI了

Brunt1104 发表于 2023-11-16 16:23

大头吃小头发表于 2023-11-16 16:18
是的，pve7.4

老哥你是什么主板？

大头吃小头 发表于 2023-11-16 16:28

Brunt1104 发表于 2023-11-16 16:23
老哥你是什么主板？

鸡架b550小雕

x2 3.0上4t系统盘 x4 4.0上了asm1166扩展6sataasm1166直接直通给黑裙

网卡两个直通给ikuai

主板自带的四个sata应该是没开iommu的原因，没见到直通设备

Brunt1104 发表于 2023-11-16 16:37

大头吃小头发表于 2023-11-16 16:28
鸡架b550小雕

x2 3.0上4t系统盘 x4 4.0上了asm1166扩展6sataasm1166直接直通给黑裙

看来可能是系统问题了，我换PVE试试吧

页: [1]

Chiphell - 分享与交流用户体验's Archiver

AMD主板的几个疑问