I have been using Debian unstable on this machine for over two years
with MATE.  In 6.10, resume worked successfully, and so did hibernation
(I have turned off secure boot for that reason).  Since upgrading to
6.12, both suspend and hibernate fail to resume: they both display the
background of my desktop, but the keyboard, mouse, and trackpoint are
unusable and unresponsive.  I tried 6.13 to see if it was fixed, and it
was not.

I will note that I usually run this machine plugged into a CalDigit TS3
Plus dock such that I use two external monitors and the machine is run
in clamshell mode with the internal display off.  If I suspend or
hibernate in such a state, it still hangs on resume, but without the
desktop background showing.

I'll mention that I'm up to date on the firmware updates, so that
doesn't fix it.  Also, switching between S3 and whatever the non-S3
(Windows-supported) option is in the firmware setup doesn't matter, and
it fails both ways.  I've also noticed that memory encryption being
enabled or disabled doesn't seem to affect it, either.

I don't believe this is a MATE bug because it has worked for some time
with MATE, so I assume this is a kernel bug.  I have traditionally
rebooted only infrequently and mostly suspended or hibernated, so it's
hard for me to say when this first started happening.  I'm filing this
because it is annoying (I need to use my dock for my work machine during
the day and want to switch back at night, so I now have to reboot) and
because it will also affect trixie and anyone using it, and ThinkPad X1
Carbons are a popular Linux laptop.

Hi Brian,

So it's obviously okay that you fill this report and understand it's
annoying. But we likely need your help here :)

So you know it worked with 6.10 but latest in 6.12 not, can you first
identify by fetching the old linux-images (use the snapshots.d.o
service) to pinpoint the Debian revision ranges where the regression
is introduced?

What is that range? Could you next bisect. Assuming we get even a
result within one stable series, this would be great, because then
next I would like to ask if you can bisect the respective upstream
changes to the breaking commits (still works if the issue is
introduced on major version change, then bisect might take longer).

Can you do that and report back around which change the hehaviour
regressed?

Regards,
Salvatore

Yeah, I can do that.  It might be this weekend before I finish, but I
can pin it down to some versions.  I'll get started on a few tonight.

I am less sure that I can build and install a functional kernel by hand.
I have no problem bisecting a change (I contribute to Git, after all),
but I don't usually do custom kernel builds.  I'll see if I can coerce
the linux package to build a custom version from upstream.

I've done some testing and I've found that 6.10.12 is the last good
version and 6.11-rc4 is the first bad version.  The former resumes
correctly from suspend and hibernate and the latter fails in both cases.
I also tested suspend on 6.11.2 and 6.11.10 and they also failed.

In all cases, to test suspend, I booted the machine on the kernel in
question, logged in, waited for Firefox (which is loaded as part of my
session) to finish loading the snapshot.debian.org page, then suspended
by closing the lid.  To test hibernate, I went to the MATE shutdown
menu and chose "Hibernate" instead of closing the lid.  Resuming from
hibernation was done by booting the default kernel (which is presently
6.13.6 from experimental), although of course that kernel is overwritten
by the one from the hibernation.

One interesting detail which may or may not be relevant is that suspend
on 6.11 takes much longer, around 5 to 10 seconds before the light on
the lid begins to pulse.  It is much faster on 6.10.12.

I'm including all of the kernel logs from my tests today (and from
intermediate boots to install new kernels) as a gzipped attachment
(firewall logs excluded for privacy and because they aren't
relevant to this matter).

Hi Brian,

Thanks.

The easiest thing would be to build up a deb package of the build and
so use the bindeb-pkg target to build a binary package.

Is the following helping?
https://wiki.debian.org/DebianKernel/GitBisect

Regards,
Salvatore

Hi Brian,

Okay that is already great, thank you for the time.

Now the next would in this case be ideally to bisect mainline between
v6.10 and v6.11-rc4 to identify the commit. On each step to build a
deb package for the kernel are hilighted at:
https://wiki.debian.org/DebianKernel/GitBisect

This will be somehow time intensive until we get to the breaking
commit, but right now I see it as only way to see where it regressed
in upstream and to followup with them.

Does this help?

Regards,
Salvatore

These directions were very helpful in compiling a kernel.
Unfortunately, it led to some unusual results.

The first step in the bisect was to try the merge base, which was 6.10,
which was bad, at which point Git refused to continue because the
problem had been fixed on the v6.10 branch.  So I thought I'd try
Debian's 6.9.2, which worked.  Then I tried upstream 6.9, which also
failed.  To verify, I tried upstream 6.9.2, which failed as well.

Note that I used `make oldddefconfig` with Debian's config to configure
the kernel, stepping back from v6.11-rc4, so it's nearly identical to
the Debian kernel (it's not signed and it doesn't have debug info, but
otherwise it should be pretty much the same.  In all cases, the tooling
is whatever's in Debian unstable at the moment.

My conclusion is thus that Debian includes some patch in the 6.9 and
6.10 series that fixes suspend and hibernate, but that patch is not
included in 6.11 and newer.

I will point out that I have a very similar ThinkPad X1 Carbon Gen 10
for work running Ubuntu 24.04 and their 6.8 kernel, and suspend and
resume do work there (the machine is using Secure Boot, so I cannot
speak to hibernation).  I can try booting off a live Ubuntu 24.10 image
(which contains a 6.11 kernel) or a daily build of Ubuntu 25.04 (which
contains a 6.14 kernel) and report back if you think that would be
helpful, say, in isolating an appropriate patch to apply.  I mention
this mostly because I know they do routine testing for things like
suspend and resume on some ThinkPads, so presumably things do work
there.

Of course, if you've come to a different conclusion or have other
suggestions about how to identify a cause, I'm all ears.

Hello brian,

Can you try a vanilla 6.10.12 to confirm that there was a fix on the
v6.10.x branch? If so, it would be interesting to know which commit
fixed the issue for you. You can bisect that; I'd recommend something
like

	git bisect start --term-new=fine --term-old=broken v6.10.12 v6.10

to reduce the confusion.

Huh, this is surprising. Can you try to import the Debian patchstack
from 6.9.2 onto 6.9.2 and bisect in that one to find the fixing commit?

Using `make oldddefconfig` is unstable when going up and down in the git
history. A more suiteable approach is:

	cp .config arch/x86/configs/brian_defconfig

and then use `make brian_defconfig` instead of `make oldconfig` in each
step.

I look forward to your test results. Your findings are really strange.

Best regards
Uwe

Hi Brian,

Did you had a chance to do further investigation here?

Regards,
Salvatore

Not sure if this will work, I have never used a mailing list before. I
am seeing the same problem, my thinkpad x1 carbon gen10 failing to
resume from suspend in all kernel versions above 6.10, 6.6 is my safest
version that I have been stuck on ever since.

However, I myself am not using Debian, I am using NixOS. I only reply
here because this is the only place I have found this bug already
talked about on the internet and I don't know how/where a more
appropriate place would be.

Let me know if theres anything I can do to help get this fixed, its
quite frustrating being locked out of all newer kernels.

Thanks!

Hello Rylee,

Is 6.10.12 also problematic for you?

In don't know how installing a custom kernel on NixOS works. But if you
do (or you want install Debian :-), the test that I asked the original
reporter to do should work for you, too.

Unless 6.10.12 is also good for you, your version range looks a bit
different than Brian's. My recommendation for you would be:

	git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
	cd linux
	cat /proc/config.gz > .config
	yes '' | make oldconfig
	make localmodconfig
	make savedefconfig
	cp defconfig arch/x86/configs/tralala_defconfig

	git bisect start v6.11 v6.6

And then iterate:

	make tralala_defconfig
	make
	make install

and test the newly installed kernel. Depending on the kernel having the
problem or not run:

	git bisect good

or

	git bisect bad

and repeat until git identifed the problematic commit.

(Note: Make sure to pick the right kernel to boot, it's not always the
one with the highest version number. After a test you can remove the
installed kernel at any time (and you probably want that to not run out
of disk space).)

(Note #2: It's only a guess from me that `make install` works on NixOS,
in doubt consult their documentation.)

Best regards
Uwe

Hi Uwe,

I finally got around to running a bisect, I started with confirming
that the tag v6.6 worked (that there weren't any NixOS patches making
the NixOS 6.6 build work) and then bisected from v6.6 as the good and
v6.11 as the bad, these were the results:

```
16:33:41 > git bisect bad
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[4812509e916bd79a17708de8371a94265f47a7bf] minixfs: use
offset_in_page()

16:36:56 > git bisect log
git bisect start
# status: waiting for both good and bad commits
# good: [ffc253263a1375a65fa6c9f62a893e9767fbebfa] Linux 6.6
git bisect good ffc253263a1375a65fa6c9f62a893e9767fbebfa
# status: waiting for bad commit, 1 good commit known
# bad: [98f7e32f20d28ec452afb208f9cffc08448a2652] Linux 6.11
git bisect bad 98f7e32f20d28ec452afb208f9cffc08448a2652
# skip: [aa7d6513d68bad539142f9d6c3e2faa629bc27d8] Merge tag 'tag-
chrome-platform-firmware-for-v6.9' of
git://git.kernel.org/pub/scm/linux/kernel/git/chrome-platform/linux
git bisect skip aa7d6513d68bad539142f9d6c3e2faa629bc27d8
# bad: [91d80986d13b7241f162be01a54d89fd7332d15a] wifi: iwlwifi: mvm:
move phy band to nl80211 band helper
git bisect bad 91d80986d13b7241f162be01a54d89fd7332d15a
# good: [c4101e55974cc7d835fbd2d8e01553a3f61e9e75] Merge tag 'soc-dt-
6.8' of git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc
git bisect good c4101e55974cc7d835fbd2d8e01553a3f61e9e75
# bad: [c3874bbec942dad35768d9f2e7418d207fb75844] Merge tag 'rxrpc-
iothread-20240305' of
git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs
git bisect bad c3874bbec942dad35768d9f2e7418d207fb75844
# skip: [bd736f38c014ba70ba7ec3bdc6af6fe5368d6612] Merge tag 'tty-6.8-
rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty
git bisect skip bd736f38c014ba70ba7ec3bdc6af6fe5368d6612
# skip: [7b829f6dd638c2cb45c7710bc7cd1d0395ea9bc1] drm/xe/guc: Convert
GuC registers to REG_FIELD/REG_BIT
git bisect skip 7b829f6dd638c2cb45c7710bc7cd1d0395ea9bc1
# skip: [242d18514149d86b431b6f5db5a33579ea79ebad] selftests/bpf: Fix
the u64_offset_to_skb_data test
git bisect skip 242d18514149d86b431b6f5db5a33579ea79ebad
# skip: [55d8ac9631aaa8ae3794341c52009f635a0d3188] drm/xe: make kobject
type struct as constant
git bisect skip 55d8ac9631aaa8ae3794341c52009f635a0d3188
# skip: [80955ae955d15ea5c11d55cd50032a5243a6dfd6] Merge tag 'driver-
core-6.8-rc1' of
git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core
git bisect skip 80955ae955d15ea5c11d55cd50032a5243a6dfd6
# skip: [5009d554e0d501741de1411db797a593a6fa94bb] drm/xe: Fix
xe_exec_queue_is_idle for parallel exec queues
git bisect skip 5009d554e0d501741de1411db797a593a6fa94bb
# bad: [ee0d27c90777da4c1da633aa7b91dbafd176c0c4] minixfs: change the
signature of dir_get_page()
git bisect bad ee0d27c90777da4c1da633aa7b91dbafd176c0c4
```

Hopefully this helps!

Rylee

Hello Rylee,

The bisection isn't done here yet, to properly complete it you also need
to test this last commit, i.e. 4812509e916bd79a17708de8371a94265f47a7bf
(but see below).

But even without this last test the result is surprising because the
first bad commit is one of these two:

	4812509e916b ("minixfs: use offset_in_page()")
	b85ea95d0864 ("Linux 6.7-rc1")

which isn't very plausible.

Can you please retest the following three commits:

	ee0d27c90777da4c1da633aa7b91dbafd176c0c4
	b85ea95d086471afb4ad062012a4d73cd328fa86
	4812509e916bd79a17708de8371a94265f47a7bf

? If your bisection was done correctly (and the problem is bisectable)
ee0d27c90777da4c1da633aa7b91dbafd176c0c4 is bad and
b85ea95d086471afb4ad062012a4d73cd328fa86 is good.

Best regards
Uwe

AFAICT the problem lies within the INTEL La Jolla Cove Adapter modules:
gpio_ljca
i2c_ljca
spi_ljca
usb_ljca

If you blacklist them and update your initramfs, hibernate and suspend
should work again.

create file /etc/modprobe.d/blacklist-ljca.conf with the following content:

blacklist gpio_ljca
blacklist i2c_ljca
blacklist spi_ljca
blacklist usb_ljca
blacklist mei_vsc_hw

,then update your initramfs:

sudo update-initramfs -u -k $(uname -r)

I don't know if the modules serve any purpose on debian, are broken, or if
they fail to load any firmware. At this point I don't care.

It's supposed to support Intel Vision Sensing Controller(IVSC) on Intel
Alder Lake platforms.

Let me know if this fixes your problem!

Kind regards,

Kurt

Hi everyone once more,

this seems a known issue, please see:

https://github.com/intel/ivsc-driver/issues/32

Kind regards,

Kurt

Hello Rylee,

Did you had a further chance to test those as Uwe was indicating?

Regards,
Salvatore