Dear Maintainer,

Possibly related bug [1]. Issue is similar to what is explained in this bug
but is not addressed by the fix which is already present in src:libvirt 7.6
version.

PS: Though I reporting from unstable machine actual test was done using libvirt 7.6
from unstable built for Bullseye.

I'm defining the network device pool which looks like below

<network>
  <name>passthrough</name>
  <uuid>f152e522-96d1-4a74-8aae-01f94244f8df</uuid>
  <forward mode='hostdev' managed='yes'>
    <pf dev='ens6np0'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x00' function='0x1'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x00' function='0x2'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x00' function='0x3'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x00' function='0x4'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x00' function='0x5'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x00' function='0x6'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x00' function='0x7'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x01' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x01' function='0x1'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x01' function='0x2'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x01' function='0x3'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x01' function='0x4'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x01' function='0x5'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x01' function='0x6'/>
    <address type='pci' domain='0x0000' bus='0x18' slot='0x01' function='0x7'/>
  </forward>
</network>

And the network configuration in libvirt domain looks like below

    <interface type='network'>
      <mac address='52:54:00:e1:5b:95'/>
      <source network='passthrough'/>
      <teaming type='transient' persistent='ua-backup0'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </interface>

When I start the domain even though domain starts fine VF pass through does not happen and the following
message is seen in the dmesg output

[11236.601474] audit: type=1400 audit(1630925018.676:49): apparmor="DENIED" operation="open" profile="libvirt-e70e9c2c-110c-401c-982f-cb384d158471" name="/dev/vfio/315" pid=5929 comm=43505520382F4B564D requested_mask="wr" denied_mask="wr" fsuid=64055 ouid=64055

and passthrough does not happen.

Note that this does not happen if the device pool interface is not present during start of domain and hot
attached using below command

sudo virsh attach-device --live --config debian10 network-pool-debian10.xml

To get the above working here is what I did I edited the /etc/apparmor.d/libvirt/libvirt-e70e9c2c-110c-401c-982f-cb384d158471 to
add line /dev/vfio/vfio rw, and this is what the changed file looks like

iaas@515-21020200100006:~$ sudo cat /etc/apparmor.d/libvirt/libvirt-e70e9c2c-110c-401c-982f-cb384d158471
#
# This profile is for the domain whose UUID matches this file.
#

#include <tunables/global>

profile libvirt-e70e9c2c-110c-401c-982f-cb384d158471 flags=(attach_disconnected) {
  #include <abstractions/libvirt-qemu>
  #include <libvirt/libvirt-e70e9c2c-110c-401c-982f-cb384d158471.files>
  #
  # for vfio hotplug on systems without static vfio (LP: #1775777)
  /dev/vfio/vfio rw,
}

Post the change I did following

sudo aa-teardown
sudo systemctl restart libvirtd
sudo systemctl restart apparmor

And on the next start device passthrough happens. I'm not sure if what I did is right but this seems to work
and I would be happy to see this done in the apparmor profile shipped by libvirt.

PS: I'm noob with apparmor all I did was bit of experiment to get the things working for my usecase.

If any other information is needed from my side please let me know.


[1] https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1775777


Thanks and Regards,
Vasudev

Hi Again,

Vasudev Kamath <vasudev@debian.org> writes:

Just  wanted to add that this failure happens only with device pool
pass through which is handled by the libvirt. [1]. Normal hostdev pass
through which looks like below works just fine and apparmor does not
cause issue in this case.

    <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:e1:5b:95'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x18' slot='0x00' function='0x1'/>
      </source>
      <teaming type='transient' persistent='ua-backup0'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </interface>


[1] https://libvirt.org/formatnetwork.html

Best Regards,
Vasudev

This bug still seems to be present in Trixie.  Looking at the contents of the
libvirt-<uuid>.files which is generated by libvirt, it's missing the necessary
PCI permissions when an SR-IOV virtual function is assigned via a network
instead of via a static PCI device address.

Specifically in this case, the following are missing from the permissions file:

"/sys/devices/pci0000:00/0000:00:1d.0/0000:03:10.1/resource3" rw,
"/sys/devices/pci0000:00/0000:00:1d.0/0000:03:10.1/vendor" rw,
"/sys/devices/pci0000:00/0000:00:1d.0/0000:03:10.1/reset" rw,
"/sys/devices/pci0000:00/0000:00:1d.0/0000:03:10.1/resource" rw,
"/sys/devices/pci0000:00/0000:00:1d.0/0000:03:10.1/device" rw,
"/sys/devices/pci0000:00/0000:00:1d.0/0000:03:10.1/resource0" rw,
"/sys/devices/pci0000:00/0000:00:1d.0/0000:03:10.1/config" rw,
"/dev/vfio/vfio" rw,
"/dev/vfio/[0-9]*" rw,

but the following is erroneously present:

"/dev/vhost-net" rw,


VM definition fragment which produces the behaviour:

    <interface type='network'>
      <mac address='52:54:00:e9:4b:bc'/>
      <source network='i350port1passthrough'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </interface>

Network definition:

<network>
  <name>i350port1passthrough</name>
  <uuid>389297b8-0f92-4523-9ad8-5d633cdddccf</uuid>
  <forward mode="hostdev" managed="yes">
    <pf dev="envftest"/>
  </forward>
</network>



Assigning via the pool is desirable because the PCIe devices are dynamically
assigned, so the bus slot assignments etc. are not guaranteed to persist.
Additionally live migration can fail unless the PCI device addresses are
identical on the source and destination hardware.

More details can be found in e.g. this article:

https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-pci_devices-pci_passthrough#sect-PCI_devices-PCI_passthrough