Hi!

I have recently upgraded my kernel, through bookworm backports, from
6.9.7-1~bpo12+1 to 6.10.6-1~bpo12+1, and since then I'm getting weird
GPU bugs. This  manifests itself in a few different ways: Firefox
can't render PDFs correctly, GitHub's status bar is all whacky, it's
weird. Those are irritants I can typically live through, and would
normally blame on Firefox or "the web", but more serious issues have
come up that compel me to file this bug report.

I believe there's a regression in the i915 driver that came with
6.10. According to this article:

https://www.phoronix.com/news/Linux-6.10-Intel-Xe-DRM-Patches

"Intel Has Many Improvements For The Xe Graphics Driver In Linux
6.10"... So I suspect I might have tripped on one of
those... uh... "improvements".

I've noticed, since the upgrade, that some tasks would entirely hang
my desktop environment (Sway, so Wayland). This would take half a
dozen seconds to resolve, then things would return to normal. dmesg
would look like this then:

2024-09-10T14:13:35.279267-04:00 angela kernel: [15590.895001] i915 0000:00:02.0: [drm] GPU HANG: ecode 12:1:85df9ff3, in Renderer [5331]
2024-09-10T14:13:35.279278-04:00 angela kernel: [15590.895007] i915 0000:00:02.0: [drm] Renderer[5331] context reset due to GPU hang
2024-09-10T14:15:05.196988-04:00 angela kernel: [15680.812788] i915 0000:00:02.0: [drm] GPU HANG: ecode 12:1:84df9ff6, in Renderer [5331]
2024-09-10T14:15:05.197000-04:00 angela kernel: [15680.812801] i915 0000:00:02.0: [drm] Renderer[5331] context reset due to GPU hang
2024-09-10T14:15:30.294244-04:00 angela kernel: [15705.909903] i915 0000:00:02.0: [drm] GPU HANG: ecode 12:1:87d79ff6, in Renderer [5331]
2024-09-10T14:15:30.294259-04:00 angela kernel: [15705.909915] i915 0000:00:02.0: [drm] Renderer[5331] context reset due to GPU hang
2024-09-10T14:15:37.963951-04:00 angela kernel: [15713.579779] i915 0000:00:02.0: [drm] GPU HANG: ecode 12:1:85dffdfb, in Renderer [5331]
2024-09-10T14:15:37.963964-04:00 angela kernel: [15713.579791] i915 0000:00:02.0: [drm] Renderer[5331] context reset due to GPU hang

Note that I redact nftable log lines that intermingle in the dmesg
here, which also show the rest of the kernel is alive during those
hangs.

I remember this moment well: I was in a videoconferencing call and was
browsing around complex Grafana dashboards. My fan was spinning and I
thought this was just some CPU overload. The entire display would
freeze, and my clock (which normally steps ahead once a second) would
be stuck in time. This would last 5-10 seconds, I would estimate, but
I didn't measure it with a wall clock as I was really busy that day.

But this morning, right after opening maps.google.com in Firefox, I
had a full and complete GPU crash. There are 500+ lines in dmesg for
that, which I included in the "Kernel log" section below for
everyone's sanity. Key excerpts:

2024-09-12T11:08:18.769812-04:00 angela kernel: [115197.412340] i915 0000:00:02.0: [drm] *ERROR* GT0: GUC: Engine reset failed on 0:0 (rcs0) because 0x00000000
2024-09-12T11:08:18.769824-04:00 angela kernel: [115198.012966] i915 0000:00:02.0: [drm] *ERROR* vdbox0: timed out waiting for forcewake ack request.
2024-09-12T11:08:18.769826-04:00 angela kernel: [115198.012970] i915 0000:00:02.0: [drm:add_taint_for_CI [i915]] CI tainted:0x9 by fw_domains_get_with_fallback+0x203/0x260 [i915]

Start of the crash. I assume this corresponds to a full display freeze
I experienced around that time. This gets repeated for a couple of
seconds, then this comes up:

2024-09-12T11:08:22.429388-04:00 angela kernel: [115201.672687] i915 0000:00:02.0: [drm] *ERROR* Atomic update failure on pipe C (start=19143 end=19179) time 600825 us, min 2146, max 2159, scanline start 547, end 647

... and the earlier messages start looping again, for about 30 more
seconds, then we get a kernel stall:

2024-09-12T11:08:40.427933-04:00 angela kernel: [115219.670223] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
2024-09-12T11:08:40.427943-04:00 angela kernel: [115219.670254] rcu: 	7-....: (34 ticks this GP) idle=74e4/1/0x4000000000000000 softirq=903226/903226 fqs=1787
2024-09-12T11:08:40.427944-04:00 angela kernel: [115219.670268] rcu: 	(detected by 13, t=5252 jiffies, g=8296709, q=11498 ncpus=16)
2024-09-12T11:08:40.427945-04:00 angela kernel: [115219.670272] Sending NMI from CPU 13 to CPUs 7:
2024-09-12T11:08:40.427945-04:00 angela kernel: [115219.670278] NMI backtrace for cpu 7

then loop again for 10 seconds, another stall, loop again for 30
seconds, then a watchdog lockup:

2024-09-12T11:09:14.094726-04:00 angela kernel: [115253.337990] watchdog: BUG: soft lockup - CPU#7 stuck for 53s! [kworker/u64:10:411461]

loop again for 10 seconds, then a fence expiration thing, presumably
from the DRM scheduler:

2024-09-12T11:09:25.510674-04:00 angela kernel: [115264.753625] Fence expiration time out i915-0000:00:02.0:sway[4388]:acdacc!

this loops around real fast, then 2 seconds later:

2024-09-12T11:09:27.312474-04:00 angela kernel: [115265.354219] i915 0000:00:02.0: [drm] *ERROR* vdbox0: timed out waiting for forcewake ack request.
2024-09-12T11:09:27.312491-04:00 angela kernel: [115265.354223] i915 0000:00:02.0: [drm:add_taint_for_CI [i915]] CI tainted:0x9 by fw_domains_get_with_fallback+0x203/0x260 [i915]

... loops for another two seconds, then we get a *similar* GPU HANG
from earlier, but this one looks much more critical, as resets fail:

2024-09-12T11:09:29.178728-04:00 angela kernel: [115268.420455] i915 0000:00:02.0: [drm] GPU HANG: ecode 12:1:849ffffc, in Renderer [5331]
2024-09-12T11:09:29.529109-04:00 angela kernel: [115268.768249] i915 0000:00:02.0: [drm] *ERROR* GT0: vcs0 reset request timed out: {request: 00000001, RESET_CTL: 00000000}
[...]
2024-09-12T11:09:29.644078-04:00 angela kernel: [115268.887409] i915
0000:00:02.0: [drm] *ERROR* GT0: Failed to reset chip

Finally everything collapses a minute or two later:

2024-09-12T11:09:31.106833-04:00 angela kernel: [115270.346116] i915 0000:00:02.0: [drm] *ERROR* Session state did not clear
2024-09-12T11:09:31.806700-04:00 angela kernel: [115271.048490] i915 0000:00:02.0: [drm] Selective fetch area calculation failed in pipe A

At this point, I think what happens is that sway completely crashes,
and systemd tries to return me to gdm3, which actually fails to talk
to the driver and display a weird "Oh no! Something has gone wrong."
black on white graphic of a sad computer, also telling me that "A
problem has occurred and the system can't recover. Please contact a
system administrator."

(I have tried to contact my nearest system administrator and realized
it was myself, oh shit. Other debian developers in #debian-quebec
actually suggested I file this issue, but i digress.)

Only a hard reboot (holding the power button on the laptop for 10
seconds) recovered from this.

I'm running Debian stable ("bookworm") on a Framework 12th generation
Intel laptop, which ships with some i915 GPU, detailed below. This
happened after a resume, if that matters, and since then I have been
able to browse google maps fine. This might be related to suspend,
therefore, but I'll make more experiments to confirm this theory.

On Thu, 2024-09-12 at 12:07 -0400, Antoine Beaupre wrote:
[...]
[...]

The i915 and xe drivers are separate.  The xe driver is still
experimental and we don't enable it.

Please test the latest unstable version (6.10.9-1) to see whether this
has been fixed.

If not, please report the bug upstream, following
<https://drm.pages.freedesktop.org/intel-docs/how-to-file-i915-bugs.html>.
We don't patch the i915 driver or drm core, so it should be OK to do so
even though you are running a distribution kernel.

Let us know the upstream bug URL so we can track it.

Ben.

The same fellow DD who suggested filing this bug helpfully found what
seems to be a similar issue:

https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/7549

... except that bug report is much older, and seems to apply to 5.x
kernels. I didn't have that issue back then, nor in 6.9, so I don't
think it's the same issue (nor is it the same symptoms).

Digging in that issue tracker, however, i do find a bunch of somewhat
similar regressions:

https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/11797
https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/11899
https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/11975

I'll also add some more data to back up my claim this is a new
regression from the upgrade:

anarcat@angela:~$ sudo grep -c GPU\ HANG /var/log/kern.log*
/var/log/kern.log:7
/var/log/kern.log.1:0
/var/log/kern.log.2.gz:0
/var/log/kern.log.3.gz:0
/var/log/kern.log.4.gz:0
anarcat@angela:~$ ls -al /var/log/kern.log*
-rw-r----- 1 root adm 2 082 950 2024-09-12 14:30 /var/log/kern.log
-rw-r----- 1 root adm 3 420 621 2024-09-10 09:54 /var/log/kern.log.1
-rw-r----- 1 root adm   285 294 2024-09-03 09:21 /var/log/kern.log.2.gz
-rw-r----- 1 root adm   386 756 2024-08-26 08:53 /var/log/kern.log.3.gz
-rw-r----- 1 root adm   248 500 2024-08-18 00:00 /var/log/kern.log.4.gz

ie. 7 hangs in the `kern.log` file, which started two days ago, and none
in the other files, going back a month.

Interestingly, the kernel was upgraded early after the kern.log.1
rotation:

Start-Date: 2024-09-04  09:30:57
Commandline: apt upgrade
Requested-By: anarcat (1000)
Install: linux-kbuild-6.10.6+bpo:amd64 (6.10.6-1~bpo12+1, automatic), linux-image-6.10.6+bpo-amd64:amd64 (6.10.6-1~bpo12+1, automatic), linux-headers-6.10.6+bpo-amd64:amd64 (6.10.6-1~bpo12+1, automatic), linux-headers-6.10.6+bpo-common:amd64 (6.10.6-1~bpo12+1, automatic)
Upgrade: linux-cpupower:amd64 (6.9.7-1~bpo12+1, 6.10.6-1~bpo12+1), linux-headers-amd64:amd64 (6.9.7-1~bpo12+1, 6.10.6-1~bpo12+1), dh-debputy:amd64 (0.1.39~bpo12+1, 0.1.48~bpo12+1), libssl3:amd64 (3.0.14-1~deb12u1, 3.0.14-1~deb12u2), linux-image-amd64:amd64 (6.9.7-1~bpo12+1, 6.10.6-1~bpo12+1), tailspin:amd64 (2.0.0+dfsg-1, 3.0.2+dfsg-1), libssl-dev:amd64 (3.0.14-1~deb12u1, 3.0.14-1~deb12u2), python3-lsprotocol:amd64 (2023.0.0-1, 2023.0.1-1~bpo12+1), libcpupower1:amd64 (6.9.7-1~bpo12+1, 6.10.6-1~bpo12+1), firefox-esr:amd64 (115.14.0esr-1~deb12u1, 115.15.0esr-1~deb12u1), openssl:amd64 (3.0.14-1~deb12u1, 3.0.14-1~deb12u2), linux-libc-dev:amd64 (6.9.7-1~bpo12+1, 6.10.6-1~bpo12+1)
End-Date: 2024-09-04  09:31:48

... but I booted in the new kernel only 4 days ago:

anarcat@angela:~$ last reboot
reboot   system boot  6.10.6+bpo-amd64 Thu Sep 12 11:12   still running
reboot   system boot  6.10.6+bpo-amd64 Tue Sep 10 09:54   still running
reboot   system boot  6.10.6+bpo-amd64 Mon Sep  9 09:24   still running
reboot   system boot  6.10.6+bpo-amd64 Sun Sep  8 12:36   still running

... so this would probably explain the lack of hangs in the older log.

Hi,

I assume the problem is still present in fresh kernel in unstable or
experimental? If so can you report it upstream as Ben suggested and
link back here the specific report?

Regards,
Salvatore

Hi Antoine,

Any news here?

Regards,
Salvatore

Since then, I upgraded to trixie and I haven't seen such bugs anymore:
doas (anarcat@angela) password:
/var/log/kern.log:0
/var/log/kern.log.1:0
/var/log/kern.log.2.gz:0
/var/log/kern.log.3.gz:0
/var/log/kern.log.4.gz:0

So maybe this is fixed? uname -a:

Linux angela 6.12.21-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.21-1 (2025-03-30) x86_64 GNU/Linux

a.

Hi Antoinie,

Thanks for your quick followup!

Okay, let's close the bug then with at least 6.12.21-1.

Regards,
Salvatore

Hi Antoinie,

Thanks for your quick followup!

Okay, let's close the bug then with at least 6.12.21-1.

Regards,
Salvatore

Hi,

Thanks for the followup on this.  I have now switched from 6.1.0-35 to 6.12.22+bpo based on thi>

I have an extensive collection of i915 DRM hangs from 6.1.x kernels.  Fingers are now crossed.

I also wanted to say that a hard-reset was not required in my experience.  I used SysRq magic t>

DMI: HP HP ZBook Firefly 14 inch G10 Mobile Workstation PC/8B41, BIOS V70 Ver. 01.01.08 05/30/2>

Regards,
Scott.