| 
 | 
 
 本帖最后由 guitarbug 于 2023-5-23 21:14 编辑  
 
问题:  
在vm中添加直通PCI设备后, pve主系统死机, 直通的设备为PCIe转SATA(ASM1061芯片) 
配置: 
CPU: E3-1225 V2 
主板: Dell 9010mt Q77(已魔改BIOS, 支持PCIe转NVME启动). 
PCIe: PCIe 16x(没插卡), PCIe 4x(PCIe转NVME卡), PCIe 1x(PCIe转SATA卡). 将卡插在PCIe 16x也是一样的现象:死机. 
PVE: 安装在PCIe转NVME里, 占用PCIe 4x插槽, 版本为: 7.3-6. 
内存: 4 * 4GB. 
 
启动参数:  
Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.102-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction  
- root@Q77:~# dmesg | grep -e DMAR -e IOMMU
 
 - [ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
 
 - [ 0.013432] ACPI: DMAR 0x00000000D7FFEDB8 0000B8 (v01 INTEL SNB 00000001 INTL 00000001)
 
 - [ 0.013459] ACPI: Reserving DMAR table memory at [mem 0xd7ffedb8-0xd7ffee6f]
 
 - [ 0.042044] DMAR: IOMMU enabled
 
 - [ 0.118620] DMAR: Host address width 36
 
 - [ 0.118621] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
 
 - [ 0.118626] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
 
 - [ 0.118628] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
 
 - [ 0.118631] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
 
 - [ 0.118632] DMAR: RMRR base: 0x000000daf77000 end: 0x000000daf9dfff
 
 - [ 0.118634] DMAR: RMRR base: 0x000000db800000 end: 0x000000df9fffff
 
 - [ 0.118636] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
 
 - [ 0.118637] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
 
 - [ 0.118638] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
 
 - [ 0.119129] DMAR-IR: Enabled IRQ remapping in x2apic mode
 
 - [ 0.266769] DMAR: No ATSR found
 
 - [ 0.266770] DMAR: No SATC found
 
 - [ 0.266771] DMAR: IOMMU feature pgsel_inv inconsistent
 
 - [ 0.266773] DMAR: IOMMU feature pass_through inconsistent
 
 - [ 0.266774] DMAR: dmar0: Using Queued invalidation
 
 - [ 0.266779] DMAR: dmar1: Using Queued invalidation
 
 - [ 0.333148] DMAR: Intel(R) Virtualization Technology for Directed I/O
 
 - [ 3.911191] i915 0000:00:02.0: [drm] DMAR active, disabling use of stolen memory
 
 - root@Q77:~#
 
 
  复制代码 
 
- root@Q77:~# dmesg | grep 'remapping'
 
 - [ 0.118638] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
 
 - [ 0.119129] DMAR-IR: Enabled IRQ remapping in x2apic mode
 
 - root@Q77:~#
 
 
  复制代码 
- root@Q77:~# find /sys/kernel/iommu_groups/ -type l
 
 - /sys/kernel/iommu_groups/7/devices/0000:00:1c.0
 
 - /sys/kernel/iommu_groups/5/devices/0000:00:1a.0
 
 - /sys/kernel/iommu_groups/13/devices/0000:02:00.0
 
 - /sys/kernel/iommu_groups/3/devices/0000:00:16.0
 
 - /sys/kernel/iommu_groups/11/devices/0000:00:1e.0
 
 - /sys/kernel/iommu_groups/1/devices/0000:00:02.0
 
 - /sys/kernel/iommu_groups/8/devices/0000:00:1c.2
 
 - /sys/kernel/iommu_groups/6/devices/0000:00:1b.0
 
 - /sys/kernel/iommu_groups/14/devices/0000:03:00.0
 
 - /sys/kernel/iommu_groups/4/devices/0000:00:19.0
 
 - /sys/kernel/iommu_groups/12/devices/0000:00:1f.2
 
 - /sys/kernel/iommu_groups/12/devices/0000:00:1f.0
 
 - /sys/kernel/iommu_groups/12/devices/0000:00:1f.3
 
 - /sys/kernel/iommu_groups/2/devices/0000:00:14.0
 
 - /sys/kernel/iommu_groups/10/devices/0000:00:1d.0
 
 - /sys/kernel/iommu_groups/0/devices/0000:00:00.0
 
 - /sys/kernel/iommu_groups/9/devices/0000:00:1c.4
 
 - root@Q77:~#
 
 
  复制代码 
- root@Q77:~# lspci
 
 - 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller (rev 09)
 
 - 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
 
 - 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04)
 
 - 00:16.0 Communication controller: Intel Corporation 7 Series/C216 Chipset Family MEI Controller #1 (rev 04)
 
 - 00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (Lewisville) (rev 04)
 
 - 00:1a.0 USB controller: Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #2 (rev 04)
 
 - 00:1b.0 Audio device: Intel Corporation 7 Series/C216 Chipset Family High Definition Audio Controller (rev 04)
 
 - 00:1c.0 PCI bridge: Intel Corporation 7 Series/C216 Chipset Family PCI Express Root Port 1 (rev c4)
 
 - 00:1c.2 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 3 (rev c4)
 
 - 00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4)
 
 - 00:1d.0 USB controller: Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #1 (rev 04)
 
 - 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a4)
 
 - 00:1f.0 ISA bridge: Intel Corporation Q77 Express Chipset LPC Controller (rev 04)
 
 - 00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04)
 
 - 00:1f.3 SMBus: Intel Corporation 7 Series/C216 Chipset Family SMBus Controller (rev 04)
 
 - 02:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
 
 - 03:00.0 Non-Volatile memory controller: Solid State Storage Technology Corporation Device 9100 (rev 03)
 
 
  复制代码 
- root@Q77:~# lspci -vvt
 
 - -[0000:00]-+-00.0 Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller
 
 - +-02.0 Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller
 
 - +-14.0 Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller
 
 - +-16.0 Intel Corporation 7 Series/C216 Chipset Family MEI Controller #1
 
 - +-19.0 Intel Corporation 82579LM Gigabit Network Connection (Lewisville)
 
 - +-1a.0 Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #2
 
 - +-1b.0 Intel Corporation 7 Series/C216 Chipset Family High Definition Audio Controller
 
 - +-1c.0-[01]--
 
 - +-1c.2-[02]----00.0 ASMedia Technology Inc. ASM1062 Serial ATA Controller
 
 - +-1c.4-[03]----00.0 Solid State Storage Technology Corporation Device 9100
 
 - +-1d.0 Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #1
 
 - +-1e.0-[04]--
 
 - +-1f.0 Intel Corporation Q77 Express Chipset LPC Controller
 
 - +-1f.2 Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode]
 
 - -1f.3 Intel Corporation 7 Series/C216 Chipset Family SMBus Controller
 
  
  复制代码 
- root@Q77:~# qm config 104
 
 - boot: order=sata0;ide2;net0
 
 - cores: 2
 
 - description: wqf%0A123456
 
 - hostpci0: 0000:02:00,pcie=1
 
 - ide2: none,media=cdrom
 
 - machine: q35
 
 - memory: 2048
 
 - meta: creation-qemu=7.1.0,ctime=1670485787
 
 - name: Ubuntu
 
 - net0: virtio=22:82:F8:E1:6C:83,bridge=vmbr0,firewall=1
 
 - numa: 0
 
 - ostype: l26
 
 - sata0: local-lvm:vm-104-disk-0,size=64G
 
 - scsihw: virtio-scsi-single
 
 - smbios1: uuid=4c074828-b68c-4b12-95b7-74ddff8374e8
 
 - sockets: 1
 
 - vmgenid: 237f068b-8811-4486-9de7-284eac69bb6e
 
 - root@Q77:~#
 
 
  复制代码 
- root@Q77:~# ./iommu.sh
 
 - Group 0:        [8086:0158]     00:00.0  Host bridge                              Xeon E3-1200 v2/Ivy Bridge DRAM Controller
 
 - Group 1:        [8086:016a] [R] 00:02.0  VGA compatible controller                Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller
 
 - Group 2:        [8086:1e31]     00:14.0  USB controller                           7 Series/C210 Series Chipset Family USB xHCI Host Controller
 
 - USB:                [1d6b:0002]                 Bus 001 Device 001                       Linux Foundation 2.0 root hub
 
 - USB:                [1d6b:0003]                 Bus 002 Device 001                       Linux Foundation 3.0 root hub
 
 - Group 3:        [8086:1e3a]     00:16.0  Communication controller                 7 Series/C216 Chipset Family MEI Controller #1
 
 - Group 4:        [8086:1502] [R] 00:19.0  Ethernet controller                      82579LM Gigabit Network Connection (Lewisville)
 
 - Group 5:        [8086:1e2d] [R] 00:1a.0  USB controller                           7 Series/C216 Chipset Family USB Enhanced Host Controller #2
 
 - USB:                [8087:0024]                 Bus 003 Device 002                       Intel Corp. Integrated Rate Matching Hub
 
 - USB:                [1d6b:0002]                 Bus 003 Device 001                       Linux Foundation 2.0 root hub
 
 - Group 6:        [8086:1e20] [R] 00:1b.0  Audio device                             7 Series/C216 Chipset Family High Definition Audio Controller
 
 - Group 7:        [8086:1e10] [R] 00:1c.0  PCI bridge                               7 Series/C216 Chipset Family PCI Express Root Port 1
 
 - Group 8:        [8086:1e14] [R] 00:1c.2  PCI bridge                               7 Series/C210 Series Chipset Family PCI Express Root Port 3
 
 - Group 9:        [8086:1e18] [R] 00:1c.4  PCI bridge                               7 Series/C210 Series Chipset Family PCI Express Root Port 5
 
 - Group 10:        [8086:1e26] [R] 00:1d.0  USB controller                           7 Series/C216 Chipset Family USB Enhanced Host Controller #1
 
 - USB:                [8087:0024]                 Bus 004 Device 002                       Intel Corp. Integrated Rate Matching Hub
 
 - USB:                [1d6b:0002]                 Bus 004 Device 001                       Linux Foundation 2.0 root hub
 
 - Group 11:        [8086:244e]     00:1e.0  PCI bridge                               82801 PCI Bridge
 
 - Group 12:        [8086:1e47]     00:1f.0  ISA bridge                               Q77 Express Chipset LPC Controller
 
 -                 [8086:1e02] [R] 00:1f.2  SATA controller                          7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode]
 
 -                 [8086:1e22]     00:1f.3  SMBus                                    7 Series/C216 Chipset Family SMBus Controller
 
 - Group 13:        [1b21:0612] [R] 02:00.0  SATA controller                          ASM1062 Serial ATA Controller
 
 - Group 14:        [1e95:9100] [R] 03:00.0  Non-Volatile memory controller           Device 9100
 
 - root@Q77:~#
 
 
  复制代码 
- 02:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02) (prog-if 01 [AHCI 1.0])
 
 -         Subsystem: ASMedia Technology Inc. ASM1062 Serial ATA Controller
 
 -         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
 
 -         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 
 -         Latency: 0, Cache Line Size: 32 bytes
 
 -         Interrupt: pin A routed to IRQ 29
 
 -         IOMMU group: 13
 
 -         Region 0: I/O ports at e050 [size=8]
 
 -         Region 1: I/O ports at e040 [size=4]
 
 -         Region 2: I/O ports at e030 [size=8]
 
 -         Region 3: I/O ports at e020 [size=4]
 
 -         Region 4: I/O ports at e000 [size=32]
 
 -         Region 5: Memory at f7d10000 (32-bit, non-prefetchable) [size=512]
 
 -         Expansion ROM at f7d00000 [disabled] [size=64K]
 
 -         Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit-
 
 -                 Address: fee002d8  Data: 0000
 
 -         Capabilities: [78] Power Management version 3
 
 -                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
 
 -                 Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
 
 -         Capabilities: [80] Express (v2) Legacy Endpoint, MSI 00
 
 -                 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us
 
 -                         ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
 
 -                 DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
 
 -                         RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
 
 -                         MaxPayload 128 bytes, MaxReadReq 512 bytes
 
 -                 DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
 
 -                 LnkCap: Port #1, Speed 5GT/s, Width x1, ASPM not supported
 
 -                         ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
 
 -                 LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
 
 -                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
 
 -                 LnkSta: Speed 5GT/s (ok), Width x1 (ok)
 
 -                         TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
 
 -                 DevCap2: Completion Timeout: Range ABC, TimeoutDis+ NROPrPrP- LTR-
 
 -                          10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
 
 -                          EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
 
 -                          FRS-
 
 -                          AtomicOpsCap: 32bit- 64bit- 128bitCAS-
 
 -                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
 
 -                          AtomicOpsCtl: ReqEn-
 
 -                 LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
 
 -                          Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
 
 -                          Compliance De-emphasis: -6dB
 
 -                 LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
 
 -                          EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
 
 -                          Retimer- 2Retimers- CrosslinkRes: unsupported
 
 -         Capabilities: [100 v1] Virtual Channel
 
 -                 Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
 
 -                 Arb:    Fixed- WRR32- WRR64- WRR128-
 
 -                 Ctrl:   ArbSelect=Fixed
 
 -                 Status: InProgress-
 
 -                 VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
 
 -                         Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
 
 -                         Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=01
 
 -                         Status: NegoPending- InProgress-
 
 -         Kernel driver in use: ahci
 
 -         Kernel modules: ahci
 
 
  复制代码 |   
 
 
 
 |