Dear Maintainer, I have a Debian Sid box running some KVM/QEMU guests and ever since the upgrade to 5.17 it is kernel panicing fairly reliably whenever I reboot the guest VMs on it. This is happening under both 5.17.3 and 5.17.6. [420450.562966] BUG: kernel NULL pointer dereference, address: 000000000000000b [420450.586155] #PF: supervisor write access in kernel mode [420450.603615] #PF: error_code(0x0002) - not-present page [420450.620787] PGD 0 P4D 0 [420450.629381] Oops: 0002 [#1] PREEMPT SMP PTI [420450.643410] CPU: 23 PID: 93711 Comm: qemu-system-x86 Kdump: loaded Not tainted 5.17.0-2-amd64 #1 Debian 5.17.6-1 [420450.677460] Hardware name: Intel Corporation T5520UR/T5520UR, BIOS S5500.86B.01.00.0061.030920121535 03/09/2012 [420450.710936] RIP: 0010:kvm_replace_memslot+0xcf/0x390 [kvm] [420450.729338] Code: 44 24 08 48 85 db 0f 84 3b 02 00 00 48 89 ea 48 c1 e2 04 48 01 da 48 8b 4a 08 48 85 c9 74 1e 48 8b 32 48 89 31 48 85 f6 74 04 <48> 89 4e 08 48 c7 02 00 00 00 00 48 c7 42 08 00 00 00 00 48 8d 54 [420450.791413] RSP: 0018:ffffad89c683fd70 EFLAGS: 00010206 [420450.808873] RAX: ffffad89c6a19058 RBX: ffff8b77f553b400 RCX: ffffad89c6a19110 [420450.832622] RDX: ffff8b77f553b400 RSI: 0000000000000003 RDI: ffffad89c6a19000 [420450.856371] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [420450.880122] R10: 0000000000000001 R11: 0000000000000000 R12: ffff8b77f553b200 [420450.903874] R13: 0000000000000000 R14: 0000000000000000 R15: ffffad89c6a19000 [420450.927626] FS: 00007f46eff3c640(0000) GS:ffff8b7dd7cc0000(0000) knlGS:0000000000000000 [420450.954525] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [420450.973699] CR2: 000000000000000b CR3: 00000001085ea005 CR4: 00000000000226e0 [420450.997452] Call Trace: [420451.005762] <TASK> [420451.012925] ? kmem_cache_alloc_trace+0x175/0x3e0 [420451.028676] kvm_set_memslot+0x2fe/0x490 [kvm] [420451.043622] kvm_vm_ioctl+0x2cb/0xd80 [kvm] [420451.057714] ? handle_mm_fault+0xb2/0x280 [420451.071175] __x64_sys_ioctl+0x82/0xb0 [420451.083775] do_syscall_64+0x3b/0xc0 [420451.095805] entry_SYSCALL_64_after_hwframe+0x44/0xae [420451.112696] RIP: 0033:0x7f46f6b13397 [420451.124723] Code: 3c 1c e8 1c ff ff ff 85 c0 79 87 49 c7 c4 ff ff ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d a9 da 0d 00 f7 d8 64 89 01 48 [420451.186798] RSP: 002b:00007f46eff3af68 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [420451.211981] RAX: ffffffffffffffda RBX: 000000004020ae46 RCX: 00007f46f6b13397 [420451.235732] RDX: 00007f46eff3b030 RSI: 000000004020ae46 RDI: 000000000000000c [420451.259482] RBP: 000055b038bce1f0 R08: 0000000000000000 R09: 00000000000c0000 [420451.283234] R10: 00000000000c0000 R11: 0000000000000246 R12: 00007f46eff3b030 [420451.306984] R13: 000000007ff40000 R14: 000055b038b69460 R15: 00000000000c0000 [420451.330738] </TASK> [420451.338184] Modules linked in: cpuid rpcsec_gss_krb5 nfsv4 dns_resolver nf_conntrack_netlink xfrm_user xfrm_algo br_netfilter bridge stp llc overlay autofs4 ip6_tables ip6t_rpfilter nft_chain_nat xt_MASQUERADE nf_nat xt_addrtype ip_tables xt_state xt_conntrack ipt_REJECT nf_reject_ipv4 nf_conntrack_tftp nf_conntrack_ftp xt_tcpudp ipt_rpfilter xt_CT nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables x_tables nfnetlink nfsd auth_rpcgss nfs_acl nfs lockd grace fscache netfs sunrpc jc42 vhost_vsock vmw_vsock_virtio_transport_common vsock vhost_net tun vhost vhost_iotlb tap ipmi_watchdog nbd squashfs loop dm_crypt dm_mod intel_powerclamp coretemp ipmi_ssif kvm_intel kvm irqbypass ghash_clmulni_intel mgag200 drm_shmem_helper drm_kms_helper aesni_intel cec crypto_simd rc_core cryptd acpi_ipmi iTCO_wdt intel_pmc_bxt intel_cstate ipmi_si drm iTCO_vendor_support evdev sg intel_uncore ipmi_devintf watchdog ioatdma i7core_edac i5500_temp ipmi_msghandler button acpi_cpufreq xfs [420451.338279] libcrc32c crc32c_generic hid_generic usbhid hid uas ata_generic usb_storage ses sd_mod enclosure t10_pi scsi_transport_sas crc_t10dif ata_piix crct10dif_generic uhci_hcd ehci_pci libata igb megaraid_sas ehci_hcd i2c_algo_bit crct10dif_pclmul dca crct10dif_common i2c_i801 usbcore scsi_mod ptp crc32_pclmul crc32c_intel i2c_smbus lpc_ich scsi_common usb_common pps_core [420451.734448] CR2: 000000000000000b
Still happens with the new v5.17.11 kernel package, in fact it may have gotten worse because at one point the box crashed shortly after starting a guest VM: [ 2256.562873] BUG: kernel NULL pointer dereference, address: 000000000000000b [ 2256.585773] #PF: supervisor write access in kernel mode [ 2256.602946] #PF: error_code(0x0002) - not-present page [ 2256.619832] PGD 0 P4D 0 [ 2256.628138] Oops: 0002 [#1] PREEMPT SMP PTI [ 2256.641880] CPU: 15 PID: 3258 Comm: qemu-system-x86 Kdump: loaded Not tainted 5.17.0-3-amd64 #1 Debian 5.17.11-1 [ 2256.675643] Hardware name: Intel Corporation T5520UR/T5520UR, BIOS S5500.86B.01.00.0061.030920121535 03/09/2012 [ 2256.708831] RIP: 0010:kvm_replace_memslot+0xcf/0x390 [kvm] [ 2256.726925] Code: 44 24 08 48 85 db 0f 84 3b 02 00 00 48 89 ea 48 c1 e2 04 48 01 da 48 8b 4a 08 48 85 c9 74 1e 48 8b 32 48 89 31 48 85 f6 74 04 <48> 89 4e 08 48 c7 02 00 00 00 00 48 c7 42 08 00 00 00 00 48 8d 54 [ 2256.788713] RSP: 0018:ffffb8dd467dfd70 EFLAGS: 00010206 [ 2256.805885] RAX: ffffb8dd468518b8 RBX: ffff8b57b7598e00 RCX: ffffb8dd46851af8 [ 2256.829350] RDX: ffff8b57b7598e00 RSI: 0000000000000003 RDI: ffffb8dd46851000 [ 2256.852815] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000860 [ 2256.876279] R10: 000000000000000b R11: 0000000000000004 R12: 0000000000000000 [ 2256.899744] R13: 0000000000000000 R14: 0000000000000001 R15: ffffb8dd46851000 [ 2256.923207] FS: 00007f85ddf10640(0000) GS:ffff8b5d97bc0000(0000) knlGS:0000000000000000 [ 2256.949816] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2256.968704] CR2: 000000000000000b CR3: 0000000675c84004 CR4: 00000000000226e0 [ 2256.992168] Call Trace: [ 2257.000193] <TASK> [ 2257.007071] ? _raw_read_unlock+0x18/0x30 [ 2257.020245] kvm_set_memslot+0x3c2/0x4a0 [kvm] [ 2257.034888] kvm_vm_ioctl+0x2cb/0xd80 [kvm] [ 2257.048674] ? handle_mm_fault+0xb2/0x280 [ 2257.061848] __x64_sys_ioctl+0x82/0xb0 [ 2257.074164] do_syscall_64+0x3b/0xc0 [ 2257.085908] entry_SYSCALL_64_after_hwframe+0x44/0xae [ 2257.102511] RIP: 0033:0x7f85e12c0397 [ 2257.114249] Code: 3c 1c e8 1c ff ff ff 85 c0 79 87 49 c7 c4 ff ff ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d a9 da 0d 00 f7 d8 64 89 01 48 [ 2257.176038] RSP: 002b:00007f85ddf0eea8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 2257.200933] RAX: ffffffffffffffda RBX: 000000004020ae46 RCX: 00007f85e12c0397 [ 2257.224398] RDX: 00007f85ddf0ef70 RSI: 000000004020ae46 RDI: 000000000000000c [ 2257.247861] RBP: 00005643173201f0 R08: 0000000000000007 R09: 00007f85d40e9ad0 [ 2257.271326] R10: 00000000fd000000 R11: 0000000000000246 R12: 00007f85ddf0ef70 [ 2257.294791] R13: 0000000001000000 R14: 0000564317dac470 R15: 00000000fd000000 [ 2257.318258] </TASK> [ 2257.325420] Modules linked in: nf_conntrack_netlink xfrm_user xfrm_algo br_netfilter bridge stp llc overlay autofs4 ip6_tables ip6t_rpfilter nft_chain_nat xt_MASQUERADE nf_nat xt_addrtype ip_tables xt_conntrack ipt_REJECT nf_reject_ipv4 nf_conntrack_tftp nf_conntrack_ftp xt_tcpudp ipt_rpfilter xt_CT nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables x_tables nfnetlink nfsd auth_rpcgss nfs_acl nfs lockd grace fscache netfs sunrpc jc42 vhost_vsock vmw_vsock_virtio_transport_common vsock vhost_net tun vhost vhost_iotlb tap ipmi_watchdog nbd squashfs loop dm_crypt dm_mod intel_powerclamp coretemp kvm_intel ipmi_ssif kvm irqbypass mgag200 ghash_clmulni_intel drm_shmem_helper drm_kms_helper aesni_intel cec crypto_simd acpi_ipmi cryptd rc_core ipmi_si intel_cstate iTCO_wdt intel_pmc_bxt iTCO_vendor_support ipmi_devintf drm evdev intel_uncore watchdog sg ipmi_msghandler ioatdma i5500_temp i7core_edac button acpi_cpufreq xfs libcrc32c crc32c_generic uas hid_generic sd_mod ses t10_pi [ 2257.325496] usbhid enclosure usb_storage crc_t10dif hid scsi_transport_sas crct10dif_generic ata_generic uhci_hcd ehci_pci megaraid_sas igb ata_piix ehci_hcd libata i2c_algo_bit dca crct10dif_pclmul crct10dif_common scsi_mod ptp i2c_i801 crc32_pclmul usbcore crc32c_intel i2c_smbus lpc_ich scsi_common usb_common pps_core [ 2257.706780] CR2: 000000000000000b
I found a matching issue on the Arch Linux forum: https://bbs.archlinux.org/viewtopic.php?id=276648 Which ultimately links to this discussion on one of the kernel mailing lists: https://lore.kernel.org/kvm/YnHALvjWw6E94K53@google.com/ https://lore.kernel.org/kvm/20220504001219.983513-1-seanjc@google.com/ And this commit: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d187ba5312307d51818beafaad87d28a7d939adf I haven't tested a custom build with the patch applied but I can confirm that the server that I have crashing is an older box with an Intel Xeon E5645 CPU that lacks XSAVE. And now that bug 1010916 has an updated backtrace attached its clear that it is the same issue as this one.
Control: tags -1 + moreinfo Can you try a custom build with the patch applied to report back if it fixes your issue? Cf. https://kernel-team.pages.debian.net/kernel-handbook/ch-common-tasks.html#s4.2.2 Regards, Salvatore
I did 5 reboots of the guest followed by 5 complete shutdown/startup cycles of the guest. No kernel panics occurred. Based on how frequent the crashes were before, I would say thats a succesful test.
Hi, Many thanks for the confirmation. Regards, Salvatore
We believe that the bug you reported is fixed in the latest version of linux, which is due to be installed in the Debian FTP archive. A summary of the changes between this version and the previous one is attached. Thank you for reporting the bug, which will now be closed. If you have further comments please address them to 1011168@bugs.debian.org, and the maintainer will reopen the bug report if appropriate. Debian distribution maintenance software pp. Salvatore Bonaccorso <carnil@debian.org> (supplier of updated linux package) (This message was generated automatically at their request; if you believe that there is a problem with it please contact the archive administrators by mailing ftpmaster@ftp-master.debian.org) Format: 1.8 Date: Mon, 06 Jun 2022 20:45:23 +0200 Source: linux Architecture: source Version: 5.18.2-1 Distribution: unstable Urgency: medium Maintainer: Debian Kernel Team <debian-kernel@lists.debian.org> Changed-By: Salvatore Bonaccorso <carnil@debian.org> Closes: 927252 1008933 1011168 1012298 Changes: linux (5.18.2-1) unstable; urgency=medium . * New upstream stable update: https://www.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.18.1 https://www.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.18.2 - netfilter: nf_tables: disallow non-stateful expression in sets earlier (CVE-2022-1966) - pipe: Fix missing lock in pipe_resize_ring() (ZDI-CAN-17291) - netfilter: nf_tables: sanitize nft_set_desc_concat_parse() (CVE-2022-1972) - [x86] fpu: KVM: Set the base guest FPU uABI size to sizeof(struct kvm_xsave) (Closes: #1011168) - KVM: x86: avoid calling x86 emulator without a decoded instruction (CVE-2022-1852) . [ Diederik de Haas ] * [arm64,armel.marvell] Remove duplicate MTD_SPI_NOR config option * [arm64] Remove duplicate CAN_MCP251X config option * drivers/net/can/spi: Enable CAN_HI311X as module (Closes: #927252) . [ Henning Schild ] * [x86] drivers/platform/x86: Enable SIEMENS_SIMATIC_IPC as module * [x86] drivers/leds: Enable LEDS_SIEMENS_SIMATIC_IPC as module * [x86] drivers/wdt: Enable SIEMENS_SIMATIC_IPC_WDT as module . [ Lubomir Rintel ] * [x86] Enable X86_ANDROID_TABLETS as a module . [ Michal Simek ] * [arm64] Enable Xilinx PHY driver and SI5341 clock driver . [ Zhang Ning ] * [arm64] Enable COMMON_CLK_PWM which is needed for some Amlogic SBCs * [arm64] Enable Khadas MCU and fan * [arm64] cpufreq: Enable SCPI cpufreq driver * [arm64] cpuidle: Enable CONFIG_ARM_PSCI_CPUIDLE . [ Ben Hutchings ] * drivers/firmware: Build ISCSI_IBFT as module on all architectures with ACPI. Thanks to Eric Mackay. (Closes: #1008933). * intel-iommu: Correct matching of the "intgpu_off" option value. Thanks to Markus Kolb. * random: Enable RANDOM_TRUST_BOOTLOADER. This can be reverted using the kernel parameter: random.trust_bootloader=off . [ Bastian Blank ] * [amd64] Enable X86_SGX. . [ Salvatore Bonaccorso ] * block, loop: support partitions without scanning (Closes: #1012298) * Set ABI to 1 Checksums-Sha1: bbc978531c7e2163a2289e3e0bc99e99839401a1 251540 linux_5.18.2-1.dsc a90a7730fbeee7e645054ece183fe17531c97120 131643692 linux_5.18.2.orig.tar.xz e0619b35776f131e195dac987f6b40cdd3444646 1309660 linux_5.18.2-1.debian.tar.xz 65338b5ed8b7d44ad3a7ab4f0fb6e4cb319f9f67 6456 linux_5.18.2-1_source.buildinfo Checksums-Sha256: fbff0455062e7392458992cbff2373f61b7c0073fb5ef4749c4b5ded3ae19cb8 251540 linux_5.18.2-1.dsc 1f38e1d57100000738cefce48a6e0793f40ea3483673400b66a6608548813637 131643692 linux_5.18.2.orig.tar.xz 42fadd9251319aa9be5970d2b1379406411672fba4f6d91fa493593a8a12fa2f 1309660 linux_5.18.2-1.debian.tar.xz b2653b3570fdda4bdff0138158a0e47449aba949a4089a31bc6342a280a13611 6456 linux_5.18.2-1_source.buildinfo Files: 74154bc9d12f3b4eb0c84e95f6a9fb6f 251540 kernel optional linux_5.18.2-1.dsc 382bb41286c2d066099a85868ade9414 131643692 kernel optional linux_5.18.2.orig.tar.xz 421f4510eb8c14390b1dfdf29134c435 1309660 kernel optional linux_5.18.2-1.debian.tar.xz 2de951e2c66717a70faa5ee66660db8d 6456 kernel optional linux_5.18.2-1_source.buildinfo -----BEGIN PGP SIGNATURE----- iQKmBAEBCgCQFiEERkRAmAjBceBVMd3uBUy48xNDz0QFAmKeS+xfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDQ2 NDQ0MDk4MDhDMTcxRTA1NTMxRERFRTA1NENCOEYzMTM0M0NGNDQSHGNhcm5pbEBk ZWJpYW4ub3JnAAoJEAVMuPMTQ89EaC0P/RG4h3wXhJLuHkx+lREXUpG7T4xcivco rfxGqun9FJxVOEAnbkzdUH4aDvUNJ83/sH+1k1kUYpjyzp+6hiiPkRUd1WJYmlXf sVvkWJIs25yxNtzCeLm419vPwS2gMCnLgg/WjsLr2Yl59ef/KIedxDDzdcdYBigG D6yR7ecIRAq1Fuiuv1YcpEUb+TC1TwF1mjiOTOkyqCyR+4Bi9/Lg1GoTIzDfhqUe K4As4o7XU85c/bACc/UoVrtOWLE+hQ7X9tZWOfr8CKLEOyy+pDO2GJPmUcLLpl0C 5qcz44+gQigWXMuZ1cG2cN8L8By5hJWKGBb9ny471ElPz4/x/GynHyRz6hVoaeet 1Pgyexi4Rcoc1CzLP+iDCMwowBP6PBSWFc/yj8/76wcgx1h6XlKJwNMMTOG0SCCO WbijgEaTgSD3f/JpB7PK9i6lfnCk8PgQXsgP2Njq8ngnLBylqFZMfyX2Mu/BChzA Nozi5zoVDqAmMvtyb5tywkdzXRYtrU8flMKCtxhi5KKmRJZYMGV4HB8nv6TY6f6f lGLPIS4H7MQaoUiQOVfMGALEF1JKm7IHaNdVadGQNV5NBXr2ksUFv+jPqQVQE1WN VtEtV1F63MlbLyh0ZTkF2LUHUCXS5pxhcuPYST4u4f2/mJvzzSo/syMf6xB7byxA DMFKZchdKO03 =P5yF -----END PGP SIGNATURE-----