Subject: linux-image-6.12.101+deb13-amd64: [REGRESSION] 6.12.101 breaks AMD SEV, SEV-ES, and SEV-SNP platform initialization Package: src:linux X-Debbugs-Cc: wizlonex@gmail.com Version: 6.12.101-1 Severity: important Hi Debian Kernel Team, Reporting a regression introduced in linux-image-6.12.101+deb13-amd64 that breaks AMD SEV/SEV-ES and SEV-SNP guest launching across different AMD EPYC platforms. Disclaimer: I am a homelab enthusiast, not a kernel developer, so I may have limited ability to answer deep technical debugging questions. However, I have tested and confirmed this regression reproducibly across two different systems with concrete logs. --- Platforms Tested --- - System 1 (AMD EPYC Rome platform - Supermicro H11DSi-NT): Running SEV-ES guests. - System 2 (AMD EPYC Milan platform): Running SEV-SNP guests.--- Observed Failure on 6.12.101+deb13-amd64 --- Host kernel command line: amd_iommu=on mem_encrypt=on kvm_amd.sev=1 kvm_amd.sev_es=1 1. Diagnostic probe via Virtee 'snphost ok' tool indicates uninitialized platform: [ PASS ] - Secure Encrypted Virtualization (SEV) [ FAIL ] - SEV-ES initialized [ FAIL ] - SEV initialized: Uninitialized 2. All SEV-ES guests fail to start with QEMU errors: libvirtd[12953]: internal error: QEMU unexpectedly closed the monitor (vm='solo-pub-ws'): qemu-system-x86_64: -accel kvm: sev_common_kvm_init: guest policy requires SEV-ES, but host SEV-ES support unavailable qemu-system-x86_64: -accel kvm: failed to initialize kvm: Operation not permitted 3. Kernel dmesg comparison indicates CCP/SEV platform init is delayed/missing at boot: On 6.12.101, 'ccp ... SEV API:0.24 build:22' does not fire during boot init, leaving the platform uninitialized when libvirt attempts guest autostart.--- Working Baseline on 6.12.95+deb13-amd64 --- Rebooting back into kernel 6.12.95+deb13-amd64 immediately resolves the issue across both platforms without any guest XML or QEMU changes: 1. 'snphost ok' reports ready: [ PASS ] - Secure Encrypted Virtualization (SEV) [ PASS ] - SEV-ES initialized [ PASS ] - SEV initialized: Initialized, currently running a guest 2. All SEV-ES guests (solo-pvt-ws, solo-pub-ws, solo-gateway) autostart and run cleanly.
Hi,
As you can reliably confirm the regression, please bisect the issue
between 6.12.95 and 6.12.101 to identify which commit(s) break. The
procedure is as follows:
git clone --single-branch -b linux-6.12.y https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
cd linux-stable
git checkout v6.12.95
cp /boot/config-$(uname -r) .config
yes '' | make localmodconfig
make savedefconfig
mv defconfig arch/x86/configs/my_defconfig
# test 6.12.95 to ensure this is "good"
make my_defconfig
make -j $(nproc) bindeb-pkg
... install the resulting .deb package and confirm problem does not exist
# test 6.12.101 to ensure this is "bad"
git checkout v6.12.101
make my_defconfig
make -j $(nproc) bindeb-pkg
... install the resulting .deb package and confirm problem exists
With that confirmed, the bisection can start:
git bisect start
git bisect good v6.12.95
git bisect bad v6.12.101
In each bisection step git checks out a state between the oldest
known-bad and the newest known-good commit. In each step test using:
make my_defconfig
make -j $(nproc) bindeb-pkg
... install, verify if problem exists
and if the problem is hit run:
git bisect bad
and if the problem doesn't trigger run:
git bisect good
. Please pay attention to always select the just built kernel for
booting, it won't always be the default kernel picked up by grub.
Iterate until git announces to have identified the first bad commit.
Then provide the output of
git bisect log
In the course of the bisection you might have to uninstall previous
kernels again to not exhaust the disk space in /boot. Also in the end
uninstall all self-built kernels again.
Regards,
Salvatore
Hi Debian Kernel Team,
With the help of Gemini to help me interpret my results, I have completed a
bisect on the upstream 6.12.y branch to pinpoint the exact commit causing
this SEV-ES launch failure regression. My journey wasn't as clean and
surgical as I would have liked (so it took longer), but I THINK I got
there.
The failure was introduced by:
Commit: bb1c84647025 ("crypto: ccp - Move SEV/SNP Platform initialization
to KVM")
Upstream commit: 3f8f0133a5fc9b32d0c308530320c3f2430ba5ab
---
### Verification & Bisect Details:
* LAST WORKING COMMIT: 8a599f4f74d4
* FIRST BROKEN COMMIT: bb1c84647025 (which landed in 6.12.97+)
* Verified on 6.12.101: Cleanly reverting commit bb1c84647025 on top of
6.12.101 immediately fixes the issue and allows SEV-ES VMs to launch and
run without errors.
---
### Failure Mechanism:
Commit bb1c84647025 removed `sev_platform_init(&args)` from
`sev_pci_init()` in `drivers/crypto/ccp/sev-dev.c` under the assumption
that KVM would handle initialization on-demand.
In practice on bare metal, the PSP firmware is left uninitialized (never
transitions to SEV_STATE_INIT at probe time). When QEMU initializes the KVM
accelerator and probes host SEV-ES capabilities, the ioctl fails with
-EPERM, producing the error:
qemu-system-x86_64: -accel kvm: sev_common_kvm_init: guest policy
requires SEV-ES, but host SEV-ES support unavailable
qemu-system-x86_64: -accel kvm: failed to initialize kvm: Operation not
permitted
Reverting bb1c84647025 restores the probe-time initialization and resolves
the regression.
Please let me know if you need any additional hardware traces or logs. I
have the time and resources to run tests if I can help - I have Milan, Rome
and even Naples (nothing newer) servers and I can try to do some trials if
it's helpful.
THANK YOU for your wonderful work. I have never rebooted a supermicro
server as much in my life as I have on this. LOL. I HOPE this is useful.
Very Respectfully,
Andrew Wilson
Hi I just wanted to check in to see if my prior information was useful in
identifying/confirming a bug? Is there anything else I can provide to
assist?
Very Respectfully,
Andrew Wilson
Hi Debian Kernel Team,
With the help of Gemini to help me interpret my results, I have completed a
bisect on the upstream 6.12.y branch to pinpoint the exact commit causing
this SEV-ES launch failure regression. My journey wasn't as clean and
surgical as I would have liked (so it took longer), but I THINK I got
there.
The failure was introduced by:
Commit: bb1c84647025 ("crypto: ccp - Move SEV/SNP Platform initialization
to KVM")
Upstream commit: 3f8f0133a5fc9b32d0c308530320c3f2430ba5ab
---
### Verification & Bisect Details:
* LAST WORKING COMMIT: 8a599f4f74d4
* FIRST BROKEN COMMIT: bb1c84647025 (which landed in 6.12.97+)
* Verified on 6.12.101: Cleanly reverting commit bb1c84647025 on top of
6.12.101 immediately fixes the issue and allows SEV-ES VMs to launch and
run without errors.
---
### Failure Mechanism:
Commit bb1c84647025 removed `sev_platform_init(&args)` from
`sev_pci_init()` in `drivers/crypto/ccp/sev-dev.c` under the assumption
that KVM would handle initialization on-demand.
In practice on bare metal, the PSP firmware is left uninitialized (never
transitions to SEV_STATE_INIT at probe time). When QEMU initializes the KVM
accelerator and probes host SEV-ES capabilities, the ioctl fails with
-EPERM, producing the error:
qemu-system-x86_64: -accel kvm: sev_common_kvm_init: guest policy
requires SEV-ES, but host SEV-ES support unavailable
qemu-system-x86_64: -accel kvm: failed to initialize kvm: Operation not
permitted
Reverting bb1c84647025 restores the probe-time initialization and resolves
the regression.
Please let me know if you need any additional hardware traces or logs. I
have the time and resources to run tests if I can help - I have Milan, Rome
and even Naples (nothing newer) servers and I can try to do some trials if
it's helpful.
THANK YOU for your wonderful work. I have never rebooted a supermicro
server as much in my life as I have on this. LOL. I HOPE this is useful.
Very Respectfully,
Andrew Wilson
Hi Andrew,
us (and this bug in the loop please). I'm asking you to forward it
directly as I expect upstream might have additional questions and/or
asking you to test a patch. So we do not need the overhead of looping
back trough us first. But we want to keep track on progress.
So as the 'culprit' is bb1c84647025 ("crypto: ccp - Move SEV/SNP
Platform initialization to KVM") make sure to include:
Ashish Kalra <ashish.kalra@amd.com>
Sean Christopherson <seanjc@google.com>
Alexey Kardashevskiy <aik@amd.com>
Herbert Xu <herbert@gondor.apana.org.au>
Additionally the stable and regression lists:
stable@vger.kernel.org
regressions@lists.linux.dev
and
Tom Lendacky <thomas.lendacky@amd.com>
John Allen <john.allen@amd.com>
"David S. Miller" <davem@davemloft.net>
linux-crypto@vger.kernel.org
linux-kernel@vger.kernel.org
Please report back here so we can link the upstream report with the
Debian bug.
Regards,
Salvatore
#regzbot introduced: 3f8f0133a5fc9b32d0c308530320c3f2430ba5ab
Hi all,
I am writing to report a regression affecting AMD SEV-ES VM launches
on the 6.12 stable branch (first noticed in 6.12.97+ / 6.12.101).
I completed a git bisect between working and failing kernels, which pointed to:
Commit: bb1c84647025 ("crypto: ccp - Move SEV/SNP Platform
initialization to KVM")
Upstream commit: 3f8f0133a5fc9b32d0c308530320c3f2430ba5ab
### Bisect & Verification Details:
* Last Known Working: 8a599f4f74d4
* First Broken Commit: bb1c84647025
* Tested on 6.12.101: Cleanly reverting bb1c84647025 immediately
resolves the issue and allows SEV-ES guests to boot normally.
### Failure Description:
With commit bb1c84647025 applied, `sev_platform_init(&args)` is
removed from `sev_pci_init()`, assuming KVM handles on-demand
initialization.
On bare metal, the PSP firmware appears to remain uninitialized (does
not reach SEV_STATE_INIT at probe time). When QEMU initializes KVM and
queries host SEV-ES capabilities, the ioctl fails with -EPERM:
qemu-system-x86_64: -accel kvm: sev_common_kvm_init: guest policy
requires SEV-ES, but host SEV-ES support unavailable
qemu-system-x86_64: -accel kvm: failed to initialize kvm: Operation
not permitted
### Hardware & Environment:
* Tested on: AMD EPYC (Rome / Milan) Supermicro platforms
* Kernel: Linux 6.12.101
* Hypervisor: QEMU / KVM
I have access to Naples, Rome, and Milan test hardware and am happy to
test any patches, provide dmesg traces, or assist with debugging. I
am not a tech-expert of any kind, but I have time to do some testing
if it helps. THANK YOU for the great work.
Original Debian bug reference:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1145026
Thanks,
Andrew Wilson
Can you try v6.12.104 or later? I'm pretty sure this is fixed by commit
6b748c39d18e ("KVM: SVM: Add support to initialize SEV/SNP functionality in KVM"),
i.e. is the same thing as:
https://lore.kernel.org/all/20260814100652.225499-1-jinpu.wang@ionos.com
Note, SNP is still buggered on 6.12.y if CONFIG_KVM_AMD=y, I'll try and send a
backport for that soonish.
https://lore.kernel.org/all/521b22fa-ef09-4449-909f-0120edfc4b24@oracle.com
Hi Sean, Salvatore,
Tested-by: Andrew Wilson <your.email@address.com>
(With strong assistance from Gemini guiding me through builds, module
DKMS rebuilds, and interpreting logs!)
I can confirm that commit 6b748c39d18e completely resolves the issue,
and SEV, SEV-ES, AND SEV-SNP are fully functional on official Debian
packages (tested on EPYC Rome and EPYC Milan servers).
Here are the test results from bare-metal hardware:
1. Upstream v6.12.104 (Source build) on AMD EPYC (Rome):
* Tested SEV-ES: Functional, VMs launch cleanly.
2. Debian 6.12.107-1 (linux-image-6.12.107+deb13-amd64) on AMD EPYC (Rome):
* Tested SEV & SEV-ES: Functional.
[ 3.676036] kvm_amd: SEV enabled (ASIDs 16 - 253)
[ 3.690542] kvm_amd: SEV-ES enabled (ASIDs 1 - 15)
[ 3.735896] ccp 0000:26:00.1: SEV API:0.24 build:22
3. Debian 6.12.107-1 (linux-image-6.12.107+deb13-amd64) on AMD EPYC (Milan):
* Tested SEV-SNP: Fully functional!
* Host log:
[ 4.678767] kvm_amd: SEV enabled (ASIDs 256 - 509)
[ 4.684694] kvm_amd: SEV-ES enabled (ASIDs 1 - 255)
[ 4.690563] kvm_amd: SEV-SNP enabled (ASIDs 1 - 255)
[ 6.212275] ccp 0000:47:00.1: SEV-SNP API:1.58 build:2
* Guest dmesg:
[ 0.908804] Memory Encryption Features active: AMD SEV SEV-ES SEV-SNP
[ 0.908820] SEV: Status: SEV SEV-ES SEV-SNP
[ 2.109524] SEV: SNP running at VMPL0.
[ 3.439649] sev-guest sev-guest: Initialized SEV guest driver
(using VMPCK0 communication key)
Because Debian builds KVM as a module (CONFIG_KVM_AMD=m), we avoid the
built-in CONFIG_KVM_AMD=y SNP race on 6.12.y, so both SEV-ES and
SEV-SNP are completely working out-of-the-box on 6.12.107-1.
Salvatore: Debian bug #1145026 can be marked fully resolved by
linux-image-6.12.107-1.
Thank you all very much!
Very Respectfully,
Andrew Wilson
Hi, Thanks for the confirmation, thanks Sean for checking the report! On Debian's side I'm thus closing the bugreport. Regards, Salvatore