#866314 linux-image-4.9.0-3-686-pae: 100+ times slower disk writes on 4.x+/i386/16+RAM, compared to 3.x

Package:
src:linux
Source:
linux
Submitter:
Holger Levsen
Date:
2022-06-01 14:45:04 UTC
Severity:
normal
Tags:
#866314#5
Date:
2017-06-28 19:53:48 UTC
From:
To:
package: linux-image-4.9.0-3-686-pae
version: 4.9.30-2+deb9u2
# severity: important ?
forwarded: https://bugzilla.kernel.org/show_bug.cgi?id=196157

Hi,

upon upgrading the i386 build nodes for tests.reproducible-builds.org to Stretch
I noticed a huge performance loss, which I could "fix" by installing
linux-image-4.9.0-3-amd64:amd64 (version 4.9.30-2+deb9u2 too).
(which btw is a nice example for Multiarchs's usefulness…)

Today Vagrant pointed me to https://bugzilla.kernel.org/show_bug.cgi?id=196157
which is the upstream issue tracking this.

I'm filing this bug for the benefit of other Debian users and for myself to
benefit from the BTS subscription and tracking features…

We want to run half of our i386 build nodes with a 32 bit kernel and the other
half with an 64 bit kernel to test reproducibility under this variation, so
I'm really looking forward to see this bug fixed soon. Other people can
probably just keep running the amd64 kernel, thus I've decided for normal
severity for this issue.

Thanks for maintaining src:linux!

#866314#12
Date:
2017-06-28 21:35:39 UTC
From:
To:
I would say only 'normal', because this isn't a sensible configuration.

[...]

Why don't you assign a smaller amount of RAM to the 32-bit VMs?  Are
there packages that need this much to build?

Ben.

#866314#17
Date:
2017-06-28 23:22:06 UTC
From:
To:
I was saying today that there are probably 100 people running i386 with i386
kernels with more than 16gb ram in the world. And then corrected myself to
"1000"…

;-)

because there are up to ten builds running simulataneously on these machines.
5 on average, and usually between 3 and 7 I'd say…

And hey, it used to work nicely.

(As a workaround I plan to use 8 machines with 15gb ram instead of 4 with 35…)

#866314#22
Date:
2017-06-29 00:12:18 UTC
From:
To:
actually this wont work here, there are not enough storage ressources to run
8 instead of 4 machines…

#866314#33
Date:
2021-05-02 09:39:05 UTC
From:
To:
Hi Holger,

Doing some maintenance on open bugs for src:linux to bring unnecessary
still open bugs down, this one appeared as well on the radar of older
bugs for older kernels.

Do you still want to keep this bug open for some reason, or should we
close it? I'm fine either way, but here I wanted to ask epxlicitly, as
there was still some movements up to february 2020 in the referenced
upstream report.

I just wonder if it will be worth off, but maybe yes.

Regards,
Salvatore

#866314#38
Date:
2021-05-03 11:26:55 UTC
From:
To:
Hi Salvatore,

I suppose it might be time to reduce memory on our i386 nodes to 15gb and
see how that goes. We're still seeing our i386 nodes hanging basically
every other day and I hope reducing memory will fix that.

I suppose it would be nice to keep this bug open but I don't care that much
either way. I understand i386 with lots of memory is super uncommon.

cc:ing Mattia for his opinion on this.

#866314#43
Date:
2022-05-31 23:27:32 UTC
From:
To:
Control: tag -1 moreinfo

What's the current status wrt this bug?

#866314#50
Date:
2022-06-01 12:09:28 UTC
From:
To:
we downgraded the memory of our nodes to 8gb to work around the issue...
#866314#55
Date:
2022-06-01 12:53:06 UTC
From:
To:
"People are still hurting from this.  It does seem a pretty major
regression for highmem machines.

I'm surprised that we aren't hearing about this from distros.  Maybe it
only affects a subset of highmem machines?

Anyway, can we please take another look at it?  Seems that we messed up
highmem dirty pagecache handling in the 4.2 timeframe."

I didn't see a mention of this bug in that upstream bug report.

A mention of "the i386 build nodes for tests.reproducible-builds.org" may also
help to increase the priority for the upstream devs.
And if it's possible to equip one of the machines with 16G again and provide
access to the upstream devs to that machine, that will likely increase the
chances that something may happen.

So it would be really helpful if this would be added to the upstream bug
report. Otherwise it's extremely unlikely anything will change.

#866314#60
Date:
2022-06-01 13:11:37 UTC
From:
To:
then please do so?!?
#866314#65
Date:
2022-06-01 14:01:10 UTC
From:
To:
And with that I've lost any motivation to further reduce the Debian kernel bug
list. Let's keep this and other pointless bug open for ever.

Well done.

#866314#70
Date:
2022-06-01 14:15:51 UTC
From:
To:
you seem to assume that I know how to report upstream bugs to the kernel.
yet I don't know.

#866314#75
Date:
2022-06-01 14:42:14 UTC
From:
To:
https://bugzilla.kernel.org/show_bug.cgi?id=196157 is a (standard) Bugzilla
instance where you could create an account if you don't have it and then add
your response to that bug.

Andrew Morton seems to prefer plain email and his' message can be found
on lore.kernel.org here:
https://lore.kernel.org/linux-mm/20170622123736.1d80f1318eac41cd661b7757@linux-foundation.org/

That message also contains instructions at the bottom how to do it (properly).

If you have further questions, I'll do my best to answer them.

HTH,
  Diederik