#1076497 incus doesn't shut down gracefully on a reboot

Package:
incus
Source:
incus
Description:
Powerful system container and virtual machine manager - daemon
Submitter:
Harald Dunkel
Date:
2024-07-23 08:57:03 UTC
Severity:
normal
#1076497#5
Date:
2024-07-17 08:56:04 UTC
From:
To:
If I reboot an incus host with 2 instances (1 LXC + 1 KVM)
running, then there are a few I/O errors on the console:

[FAILED] Failed unmounting run-user-0.mount - /run/user/0.
[FAILED] Failed unmounting var-lib-…ncus/devices/limited/config.mount.
[FAILED] Failed unmounting var-lib-…unt - /var/lib/incus/guestapi.
[FAILED] Failed unmounting var-lib-…/default/virtual-machines/limited.
[  112.573288] I/O error, dev sda, sector 214389968 op 0x0:(READ) flags 0x0 phys_seg 9 prio class 0
[  112.605377] I/O error, dev sda, sector 214390040 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  112.647117] I/O error, dev sda, sector 214390040 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0

Obviously the instances were not shut down correctly, even
though they were idle.

storage is LVM. The images have been created looking at the
online tutorial:

% incus launch images:debian/12 first
% incus launch images:debian/12 limited --vm
% incus snapshot create first first.snapshot


root@srvl051:~# incus list -v
    NAME     STATE            IPV4           IPV6       TYPE        SNAPSHOTS
   first    RUNNING  192.168.102.154 (eth0)         CONTAINER        1
   limited  RUNNING  192.168.99.118 (enp5s0)        VIRTUAL-MACHINE  0

root@srvl051:~# incus storage list -f compact
    NAME    DRIVER  DESCRIPTION  USED BY   STATE
   default  lvm                  3        CREATED

root@srvl051:~# incus storage volume list default -f compact
           TYPE                  NAME          DESCRIPTION  CONTENT-TYPE  USED BY
   container             first                              filesystem    1
   container (snapshot)  first/first.snapshot               filesystem    0
   virtual-machine       limited                            block         1

root@srvl051:~# lsblk
NAME                                            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda                                               8:0    0 14.3T  0 disk
|-sda1                                            8:1    0    4T  0 part
| |-default-IncusThinPool_tmeta                 254:0    0  128M  0 lvm
| | `-default-IncusThinPool-tpool               254:2    0    4T  0 lvm
| |   |-default-IncusThinPool                   254:3    0    4T  1 lvm
| |   |-default-containers_first                254:4    0   10G  0 lvm  /var/lib/incus/storage-pools/default/containers/first
| |   |-default-virtual--machines_limited.block 254:5    0   16G  0 lvm
| |   `-default-virtual--machines_limited       254:6    0  500M  0 lvm  /var/lib/incus/devices/limited/config.mount
| |                                                                      /var/lib/incus/storage-pools/default/virtual-machines/limited
| `-default-IncusThinPool_tdata                 254:1    0    4T  0 lvm
|   `-default-IncusThinPool-tpool               254:2    0    4T  0 lvm
|     |-default-IncusThinPool                   254:3    0    4T  1 lvm
|     |-default-containers_first                254:4    0   10G  0 lvm  /var/lib/incus/storage-pools/default/containers/first
|     |-default-virtual--machines_limited.block 254:5    0   16G  0 lvm
|     `-default-virtual--machines_limited       254:6    0  500M  0 lvm  /var/lib/incus/devices/limited/config.mount
|                                                                        /var/lib/incus/storage-pools/default/virtual-machines/limited
`-sda2                                            8:2    0    4T  0 part /export
sdb                                               8:16   0  256G  0 disk
`-sdb1                                            8:17   0  256G  0 part /


I would have expected the instances were shut down with a
reasonable timeout by default. The volumes should have been
unmounted and deactivated.

Kernel is 6.7.12 from the bookworm-backports repository.


Regards

Harri

#1076497#10
Date:
2024-07-23 08:54:38 UTC
From:
To:
Hi Harri,

  I think I've been able to reproduce the issue you're seeing by
introducing a long sleep into the /usr/libexec/incus/shutdown script.
With that I also see (briefly before restart) failures unmounting lvm
paths used by Incus.

  The incus-startup service does define a timeout of 10 minutes for
starting and stopping incus at boot/shutdown, so I would guess that
there's some sort of systemd service dependency that's not defined or
missing that allows file systems to be unmounted before Incus is able
to shutdown all its containers/VMs. I'll do some more investigation
into this issue.

Mathias