Package: e2fsprogs
Version: 1.47.0-1
Severity: grave
Justification: renders package unusable
X-Debbugs-Cc: debian-boot@lists.debian.org, grub2@packages.debian.org

(Please keep debian-boot@lists.debian.org and grub2@packages.debian.org
in the loop.)

Hi,

Spotted via debian-installer tests: grub-install fails with “unknown
filesystem” when trying to run a simple `grub-install /dev/sda` with
an all-default installation.

While there might be something wrong about e2fsprogs-udeb specifically,
possibly only affecting the installer context, I'm not sure what that
means for existing systems, hence the severity.

The “e2fsprogs gets a new upstream release and new flags” hypothesis was
confirmed by building d-i with e2fsprogs-udeb_1.46.6-1_amd64.udeb
rebranded as 1.48, which made the problem disappear.


Cheers,

It is the metadata_csum_seed feature that is now enabled by default
which grub-install does not grok.  This has been reported "recently"[1].

Cheers,
       Sven


1. https://bugs.debian.org/866603

2021:

commit 7fd5feff97c4b1f446f8fcf6d37aca0c64e7c763
Author: Javier Martinez Canillas <javierm@redhat.com>
Date:   Fri Jun 11 21:36:16 2021 +0200

    fs/ext2: Ignore checksum seed incompat feature

    This incompat feature is used to denote that the filesystem stored its
    metadata checksum seed in the superblock. This is used to allow tune2fs
    changing the UUID on a mounted metdata_csum filesystem without having
    to rewrite all the disk metadata. However, the GRUB doesn't use the
    metadata checksum at all. So, it can just ignore this feature if it
    is enabled. This is consistent with the GRUB filesystem code in general
    which just does a best effort to access the filesystem's data.

    The checksum seed incompat feature has to be removed from the ignore
    list if the support for metadata checksum verification is added to the
    GRUB ext2 driver later.

    Suggested-by: Eric Sandeen <esandeen@redhat.com>
    Suggested-by: Lukas Czerner <lczerner@redhat.com>
    Signed-off-by: Javier Martinez Canillas <javierm@redhat.com>
    Reviewed-by: Lukas Czerner <lczerner@redhat.com>
    Reviewed-by: Daniel Kiper <daniel.kiper@oracle.com>

Unfortunately, this just missed the last grub release, since grub 2.06
was tagged on June 8, 2021.  <Sigh>

There are two ways we can fix this.  One is we can disable the
checksum seed feature in the Debian release of mke2fs.conf.  Or we can
land this above-mentioned commit into the grub2 package.  Since the
hard freeze is almost upon us, I'm happy to handle this either way.

If we *are* going to backport some grub2 patches, may also make a plug
for this one, which is also in the upstream grub2 git repo:

commit 2e9fa73a040462b81bfbfe56c0bc7ad2d30b446b
Author: Theodore Ts'o <tytso@mit.edu>
Date:   Tue Aug 30 22:41:59 2022 -0400

    fs/ext2: Ignore the large_dir incompat feature

    Recently, ext4 added the large_dir feature, which adds support for
    a 3 level htree directory support.

    The GRUB supports existing file systems with htree directories by
    ignoring their existence, and since the index nodes for the hash tree
    look like deleted directory entries (by design), the GRUB can simply do
    a brute force O(n) linear search of directories. The same is true for
    3 level deep htrees indicated by large_dir feature flag.

    Hence, it is safe for the GRUB to ignore the large_dir incompat feature.

    Fixes: https://savannah.gnu.org/bugs/?61606

    Signed-off-by: Theodore Ts'o <tytso@mit.edu>
    Reviewed-by: Daniel Kiper <daniel.kiper@oracle.com>

Otherwise, what can happen is that users may choose to enable the
large_dir feature, and if they do it on the root file system, they can
end up making their system unbootable.  I've had some Arch and Gentoo
users get bitten by this....

Alternatively, I can apply this patch to e2fsprogs:

diff --git a/misc/mke2fs.conf.in b/misc/mke2fs.conf.in
index b7fc95df7..ff47bdb39 100644
--- a/misc/mke2fs.conf.in
+++ b/misc/mke2fs.conf.in
@@ -11,7 +11,7 @@
 		features = has_journal
 	}
 	ext4 = {
-		features = has_journal,extent,huge_file,flex_bg,metadata_csum,metadata_csum_seed,64bit,dir_nlink,extra_isize,orphan_file
+		features = has_journal,extent,huge_file,flex_bg,metadata_csum,64bit,dir_nlink,extra_isize,orphan_file
 	}
 	small = {
 		blocksize = 1024

Which will kind of suck, since the reason for enabling
metadata_csum_seed is to be able to reliably change the label and file
system uuid on a mounted file system, which matters for certain cloud
workloads.  Specifically, for Google's Cloud Optimized OS, which
attempts to update the file system UUID and resize the root file
system to fit the size of the cloud-emulated block device on two
separate, racing systemd unit files.  This which works 99.9% of the
time, but when you launch a huge number of cloud images, that last
0.1% of the time is really noticeable.

But that's for COS; if we have to disable the metadata_csum_seed
feature on Debian file systems, I can live with that.

Cheers,

						- Ted

Control: reassign -1 grub2

...

I've just queued these up in our repo for the next grub upload, due in
a few days.

We believe that the bug you reported is fixed in the latest version of
grub2, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 1030846@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Steve McIntyre <93sam@debian.org> (supplier of updated grub2 package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)
Format: 1.8
Date: Thu, 09 Feb 2023 01:09:00 +0000
Source: grub2
Architecture: source
Version: 2.06-8
Distribution: unstable
Urgency: medium
Maintainer: GRUB Maintainers <pkg-grub-devel@alioth-lists.debian.net>
Changed-By: Steve McIntyre <93sam@debian.org>
Closes: 1001414 1016737 1021846 1022184 1026092 1026915 1030846
Changes:
 grub2 (2.06-8) unstable; urgency=medium
 .
   [ Steve McIntyre ]
   * Fix an issue in an f2fs security fix which caused mount
     failures. Closes: #1021846. Thanks to программист некто for helping
     to debug the problem!
   * Switch build-deps from gcc-10 to gcc-12. Closes: #1022184
   * Include upstream patch to enable EFI zboot support on arm64.
     Closes: #1026092
   * grub-mkconfig: Restore umask for the grub.cfg. CVE-2021-3981
     Closes: #1001414
   * postinst: be more verbose when using grub-install to install onto
     devices.
   * /etc/default/grub: Fix comment about text-mode console.
     Fixes #845683
   * grub-install: Don't install the shim fallback program when called
     with --removable. Closes: #1016737
   * grub-install: Don't use our grub CD EFI image for --removable.
     Closes: #1026915. Thanks to Pascal Hambourg for the patch.
   * Ignore some new ext2 flags to stay compatible with latest mke2fs
     defaults. Closes: #1030846
 .
   [ Colin Watson ]
   * Remove myself from Uploaders.
Checksums-Sha1:
 234753e3ff9b53eff02161d83d6e177ac2bea2ed 7057 grub2_2.06-8.dsc
 1770e37881e16a8b08ecc321fd3d4b50eca0f771 1101464 grub2_2.06-8.debian.tar.xz
 d3e91594aa51d1aea8516811810d2769708e5619 13110 grub2_2.06-8_source.buildinfo
Checksums-Sha256:
 a466fccc73504ea37c08837eeaa7a541e8ea4745447eb497482a66c81a5c87de 7057 grub2_2.06-8.dsc
 d16aa0de7efa01a7c6fa6982b233cd1943163bb4a6502f29a039f75387494c52 1101464 grub2_2.06-8.debian.tar.xz
 3b8af19ce295e23fc048a887f2449dfbd6f2de761865a87d3e0c3bc43f1d41c8 13110 grub2_2.06-8_source.buildinfo
Files:
 40d4cb35d00c64655f6b49d5f53024f8 7057 admin optional grub2_2.06-8.dsc
 42b799b299def7faacd651b78637ab2f 1101464 admin optional grub2_2.06-8.debian.tar.xz
 41eaddffeaca3810c78a52db342d1a9f 13110 admin optional grub2_2.06-8_source.buildinfo
-----BEGIN PGP SIGNATURE-----

iQJFBAEBCAAvFiEEzrtSMB1hfpEDkP4WWHl5VzRCaE4FAmPkTaQRHDkzc2FtQGRl
Ymlhbi5vcmcACgkQWHl5VzRCaE6/LBAAlUiw6/cPFU8fIq6szxJ4rJjNjUNPdpz1
WKTY1ewjqiLg75dTuWdZeLB6rHFeC6uWXw+P/zy3P9x95PScQ6g8AgeIDaHlBL6W
m2Cy9zYpAPSBzC5mHaAjqis7YOBUUiRm7URIURrq1SODYP86w0jVP0ow5wM4PoBA
ERkkhsWUG6ZSC5lW1nGuPo30k+lrB5euBe6xYwyssFg7h+frDdx1QEIKA2Ta3fS1
aiZEM9FBIART++7Bjv1AIcjqUeRqgz3Wan6GLvtf6IJ4lYtliP39LwELgCDf43o2
6UK2/9uJ/WB5L/F1lx/kVA07+H/icXO2WVYWFLKjzLouclQFaRg3JDj69BAjr+o0
8xpafGa++WH8ItICFx/xmiyOWbQo2jUYonjk/4sd2jaIDHueLj9KszwC1KtSsuTJ
YYpjMitUf7vH+LSWztTdJ+DSPuql4my1Y+nTUBnO5T4qOXjLuiRWP6mNLHBYZfgM
r8PQLSQ2rDKBohUk9kslcchT38tx9CHLBQvwkHSu8lnQ7tAZXYtPVbHyyJrcu5Te
wO1HgnZsgRuJNZQwa6Jk/T6MnivAIZkqaz+8BoaknrbHmUeyGnslSINL7vxYxxe+
kMWdXGyGgf4JbS7f6nOvEg3EB/le9cEP/H/MRtRYtvsrEMiptdOa5B68Gy3+g7/F
FzZ5po4425c=
=CS7I
-----END PGP SIGNATURE-----

Many thanks, Steve!

					- Ted

Control: clone -1 -2
Control: reopen -2
Control: reassign -2 e2fsprogs 1.47.0-1
Control: severity -2 serious

Thanks from me as well :-).  To prevent e2fsprogs from migrating to
testing before grub2 and breaking d-i, I am reassigning a copy of this
bug back to e2fsprogs.  It may be closed once grub2 2.06-8 enters
Bookworm.   Perhaps it would also be a good idea to add a versioned
"Breaks: grub2-common (<< 2.06-8~)" to e2fsprogs.

Cheers,
       Sven

[Switching over to the cloned bug.]

That is not going to help, because IIUC grub-install is run from the
target system that you are installing, and there is no
grub2-common-udeb.

I was also thinking of the cases where the /boot filesystem is
recreated, e.g. when transferring the installation to a different disk.

Cheers,
       Sven

from pulling in an overly new version of e2fsprogs-udeb, that woul be
sufficient, no?

After all, you can install e2fsprogs 1.47, and install a newer version
of grub, and there is no problem --- unless you enable the the
csum-seed function on an already installed system.  And you can't do
that, because you it's an incompat feature.

OTOH, with e2fsprogs 1.46 you can *ready* run the command "tune2fs -O
large_dir /dev/root" on a running system, and then when you reboot,
the system won't come back, because grub will see the large_dir
feature and freak out (unecessarily).  But adding an conflict with
1.46 and grub makes no sense, since it's not _that_ likely that user
will deploy that particular foot-gun.  (It happens with Arch and
Gentoo, but those users tend to be much more adventurous, aided and
abetted by Wiki pages that encourage users to help kernel developers
beta test new features.  :-)

The reason why I really don't want to add the conflicts with e2fsprogs
1.47 is because for people who are using sid, there is aboslutely
nothing wrong with installing e2fsprogs 1.47.  It's only if they use
the installer that sucks in e2fsprogs that there's a problem --- it's
not an inherent conflict with grub per se.  If it were, then we
shouldn't allow installation of e2fsprogs 1.46 because grub2 upstream
is painfully slow in putting out releases --- and that's not
reasonable.

Now, what I *could* do is change the mke2fs.conf in e2fsprogs-udeb to
remove the default use of the csum-seed feature ---- but is it worth
it?  Hopefully the new version of grub2 will come out soon, at which
point I would then need to revert the hacked mke2fs.conf --- and your
dup'ing this bug should prevent the installer from picking up
e2fsprogs 1.47 until the new version of grub gets released.

I'll note that grub is trying to use an independent implementation of
ext4, as opposed to using the upstream libraries such as libext2 for
ext2/3/4 and libxfs for XFS, combined with grub's very slow release
cycle, means that this kind of thing is going to be happening a *lot*.
It's something that I and Darrick Wong (the XFS maintainer) have
kvetched about a number of times on our weekly conference calls.

For example, recent grub commits that impact XFS that aren't in 2.06
include:

commit a4b495520e4dc41a896a8b916a64eda9970c50ea
Author: Erwan Velu <erwanaliasr1@gmail.com>
Date:   Wed Aug 25 15:31:52 2021 +0200

    fs/xfs: Fix unreadable filesystem with v4 superblock

    The commit 8b1e5d193 (fs/xfs: Add bigtime incompat feature support)
    introduced the bigtime support by adding some features in v3 inodes....

Holding back file system development because grub2 uptsream is super
slow doesn't seem like a reasonable way forward, so I really don't
want to set this precedent.

Thinking about this some more, I'd much rather have some kind of
explicit scheme, such as:

       mkfs.xfs -T grub2-dumbdown /dev/XXX

Which disables various new file system features that aren't yet
supported by grub, but which users might very well want to use on
non-boot disks.

This could be done by doing something as simple as adding to mke2fs.conf:

[fs_types]
     grub2-dumbdown = {
	features = ^large_dir,^metadata_csum
     }

Then we could teach the Debian installer to use the file system usage
type "grub2-dumbdown".

Of course, moving forward we'd have to update mke2fs.conf as grub
finally learns new features, and since mke2fs.conf is a config file,
people would have to edit it by hand post-install if they have
customized it in any way.  Unless we add the above in
/etc/mke2fs.conf.d/grub2-dumbdown, and then teach mke2fs to understand
the /etc/mke2fs.conf.d directory...

      	     		       			  - Ted

Hi

No, it does not.  Conflicts have undefined behaviour for udebs.

Bastian

Theodore Ts'o <tytso@mit.edu> (2023-02-09):
 - installer, with anna and udebs (but that would be the same with apt
   and debs);
 - target.

There are no connections between both environments, so you can't have
package relationships in a cross-environment manner.

The installer uses whatever was available at build-time, or for
components that aren't built-in, whatever is available on the
installation image, or on the mirror. There's no “pulling in an overly
new version” that can be avoided. There's *one* version in the suite,
that's the one that's getting used, there's no alternative.

TL;DR: Some Conflicts, even if that were possible (which it is not)
       wouldn't achieve anything (except probably not offering any ext*
       option at the partioning step — not a huge win).

*sigh* @ the heavy finger pointing in various parts of your mail.
unstable, candidate for migration into testing. Until that happens,
e2fsprogs must be kept out of testing, so that the installer cannot get
a too-new e2fsprogs with a too-old grub2.

Whether we introduce a relationship between both packages making britney
wait on grub2's being ready to allow e2fsprogs to migrate alongside it,
or we keep an RC bug on e2fsprogs to keep it out of testing until grub2
gets fixed and migrated into testing… doesn't matter much to me.


I'll word it differently because I'd like to avoid more mails on this
thread: The installer pulls components for the target system from a
single distribution, there's no set of existing packages that can be
kept around (as that would be the case for existing systems using
testing or unstable), so we need packages in the distribution being
installed to be consistent.

Right now: unstable is broken; testing isn't.


[ snip stuff about grub design ]

The Bookworm freeze has started, we need to be able to install stuff, so
we need a solution for the grub2/e2fsprogs situation *now*.

I really don't care about “setting a precedent”.

[ snip brainstorming ]


Cheers,

But that problem has already been solved by cloning the bug back to
e2fpsrogs (#1030939) which will prevent e2fsprogs from transitioning
to testing, no?  So what's the problem.

The fact that grub2 upstream is super-slow in doing releases is
already a pain in *ass that has caused much pain over the years.
Adding a conflict just adds more pain, without adding any additional
benefit.  So what's the point?

     	   	   	      		 - Ted

Theodore Ts'o <tytso@mit.edu> (2023-02-10):

I never said there was a problem with the current state of things
(indeed, that's one of the two soltions I described as being equally
fine with me).

Instead, I've explained why your suggested “solution” wouldn't help,
with some context since you seemed unconvinced by previous answers from
others.


Cheers,

Hi Steve,

I believe that your fix to grub2 in Sid is not enough to handle
#1030939/#1030846.

This problem breaks e.g. vmdb2. I can no longer create a Bullseye
system image with vmdb2 on Sid, because the grub-install step in the
Bullseye chroot now fails, because the created filesystem (created with
e2fsprogs on Sid) cannot be recognized by grub. I have to downgrade
e2fsprogs to the version in Testing to get the build going again. It
also means that a Bookworm system can never be used to format and
debootstrap a Bullseye or Buster system that requires a grub
installation.

I guess that the fix applied to grub2 in Sid would have to be applied
to grub2 in Bullseye as well (and basically to any grub2 package in any
Debian or Ubuntu or Raspbian release supported by debootstrap).

This situation is really messy. It breaks basically all my image builds
with vmdb2.

Regards, Daniel

I'm confused, why does using vmdb2 on *sid* involve running
grub-install on a *bulleye* chroot?  That seems to be extremely
ill-advised.  If you are trying to install a bullseye system, why
can't you using vmdb2 on bullseye?

And if you are trying to install a sid or bookworm system using vmdb2,
why can't you (or vmdb2) use a sid or bookworm chroot for doing the
grub-install?

I can understand why we need to keep things synchronized so that a
debian installer for Bookworm be able to generate a bootable system
using the version of grub and e2fsprogs in Bookworm.

But a generalized requirement that we be able to use debootstrap and
vmdb2 to be able to bootstrap an arbitrarily old distribution doesn't
seem to be reasonable.  It would mean that we couldn't update to newer
versions of the C library, or of new file system featuers, because we
are somehow obliged to be able to boostrap ancient versions of the
system.  After all, we don't require that a binary built using
Bookworm be able to run on Bullseye.  How is this any different?

I generate test appliances for file system regression testing which
run on Debian Bullseye on my Debian Bookworm system --- and this gets
done using build chroot[1].  I even have super-convenient scripts to
create the build chroot[2].  This is not hard.... why can't vmdb2 do
the same thing?

[1] https://github.com/tytso/xfstests-bld/blob/master/Documentation/building-xfstests.md
[2] https://github.com/tytso/xfstests-bld/blob/master/setup-buildchroot

						- Ted

Am Montag, dem 13.02.2023 um 21:35 -0500 schrieb Theodore Ts'o:
(in this case using vmdb2, but I could do it manually as well). First,
one creates the paritioning and the filesystems for the target system
with the tools provided by the host system. This involves running the
unfortunate version of e2fsprogs with the breaking change. Then, one
installs the base system with deboostrap (also from the host system)
into the created partitions/filesystem. Then one (bind-)mounts the
virtual filesystems into the target systems, does a chroot into it, and
then one finishes the installation inside the chroot, including running
grub-install.

This is standard and common behavior. I have never ever seen in my
whole life someone to use grub2 from Sid to make a grub-install for a
Bullseye target. I have not even seen anybody suggest that.

Consider another example: Server providers use GRML or Bookworm with
e2fsprogs 1.47.0 as their rescue system. Now people are no longer able
to create a Bullseye system using the deboostrap method [1]. If the
host system uses e2fsprogs 1.47.0 or above and the target system for
[1] uses a grub without a fix, then this no longer works.

[1] https://www.debian.org/releases/stable/amd64/apds03

I'm sorry? I think you are completely missing the point.

Because I am using Sid and because I use features of vmdb2 which are
not available in the version in Bullseye. And this feature breaks vmdb2
and similar tools. It also breaks a standard way of installing Debian
systems.

What are you talking about? You basically break the toolchains and then
you suggest to do non-standard stuff to handle this or even avoid doing
installations of affected systems?!

You are completyl wrong. This breaks a standard way of installing any
system supported by deboostrap with a grub without a fix to deal with
this version of e2fsprogs. This isn't just about vmdb2.

What you are saying is ignorant.

If this isn't cared about, then this version of e2fsprogs shouldn't
make it into Bookworm. We are in the middle of a freeze and this breaks
toolchains and a standard way (see [1]) of installing Debian.

Daniel

...

Sorry Daniel, but I have to (mostly!) agree with Ted here. If you're
creating an image of an older release using newer tools then you'll
need to be aware that sometimes the newer tools will create things
that don't work there. If there's a bug here, I would strongly argue
that it's in vmdb2. deboostrap (for example) includes some
release-specific knowledge to cope with issues like this.

If we don't allow for this kind of change, that wouldn't allow us to
*ever* make breaking changes in some packages, and that's just not
sustainable.

Am Dienstag, dem 14.02.2023 um 10:45 +0000 schrieb Steve McIntyre:
I refer to [1]: If the host system contains the problematic e2fsprogs
and the target system doesn't contain a grub with the fix [2], then
this breaks installations. This breaks older systems *and* current
systems. For example, I neither see the necessary grub patch in both
Ubuntu 20.04 and 22.04 either. So they also cannot be installed using
the deboostrap method and the toolchain in Sid (and Bookworm if
e2fsprogs makes it there).

[1] https://www.debian.org/releases/stable/amd64/apds03
[2] Even "our" grub only contains a patch and not an upstream version
with support. So how can you even expect the target system to contain a
fix and be able to handle the created filesystem?!

Whe whole handling is completely wrong here. First, grub should have
been fixed upstream. And the change in e2fsprogs should have been done
only after "fixed" grub versions had settled. If we do it the other way
around, we have to patch grub in affected distributions as well. And
for Debian that means at least to patch Bullseye and any other release
we want to be able to install from Bookworm. I even a lot of companies
using Buster still.
ignorance shown reagarding the impact, not that fact that there is a
breaking change. And it breaks a lot! This doesn't affect just a few
minor use cases. It affects the basic way of installing a clean Ubuntu
or Debian (or derivative) on a remote server using the debootstrap
method.


And again: We are in the middle of a freeze here. And e2fsprogs pushes
a breaking change that is not even handled by any existing grub
upstream release, and is also not properly handled within Debian?!

Daniel

That's why I said *like* this, not *exactly* like this. debootstrap
has had knowledge of things like fs layouts etc. that older releases
need (e.g. merged-/usr).

Breakages happen like this, and this has happened before in similar
circumstances. If you're installing an older system using brand-new
tools, you need to be aware of the potential for things to not
work. In this particular case, all you need to do is tweak the flags
on the ext4 filesystem when you create it. This isn't that hard...

And I know of folks still working on Stretch and Jessie. How far back
do you want to tie things??

People using these tools need to be aware of the potential issue. What
would happen if you ran debootstrap with a filesystem that the target
distro doesn't know how to mount at all, for example?

Grub upstream is already known to be problematic in terms of release
cycles. We now know about this particular issue (thanks Ted!) and
we've fixed it in unstable (and soon testing).

Am Dienstag, dem 14.02.2023 um 14:58 +0000 schrieb Steve McIntyre:
[..]

This breakage is *unnecessary* and it breaks at the momnent *all*
debootstrap installations except for doing a bookworm/sid installation
themselves. That is just stupid.

Get down from your high horse and ackknowledge the problems your
behavior causes.

No it has not. You are doing a breakage on purpose during a freeze
period while the transition period is over. Do it with a proper
transition during the next release cycle.
[..]

How about staying on topic and explaining, why this transition is
necessary and has to be done the wrong way around, instead of picking
the fact that older systems still exist? You break almost *all*
installation right now. You also broke an Ubuntu 20.04 or 22.04 LTS
installation. Are they new enough?

[..]

That is different from introducing a breaking change for which no grub
upstream release has a fix yet. So basically the only system able to
handle a filesystem created with e2fsprogs 1.47.0 is Sid right now. Can
you please ignore your ego and see the problems you are causing?

You push a breaking change for no reason at all. What is the gain here
compared to the issues you are causing?

That is not enough and it is not a solution for the problem. There is
*no* grub version out there supporting this, except for the patched
version is Sid. Is this the sign for a working transition?

Which helps exactly nobody, as it still breaks all other Debian or
Ubuntu installation.

I cannot belive that you intentionally break one of the standard
methods to install Debian or Ubuntu for no reason at all and without a
proper transition. Version 1.47.0 of e2fsprogs contains nothing
necessary for Bookworm. I'll bring this to the attention of the release
team. I'm sick of your ignorance. Do a proper transition and don't
start it during a freeze.


Daniel

There is another issue with vmdb2 if you are using XFS.  Starting with
xfsprogs 5.15 (which is already in testing), bigtime is enabled by
default, so that newly created XFS file systems won't be subject to
timestamp overflow in 2038.  Grub didn't land support for this feature
until 8b1e5d1936ff ("fs/xfs: Add bigtime incompat feature support") in
May 2021, despite the fact that XFS has had this feature for years and
years and years.

So if you aren't using the latest security fixes, and you are using
XFS as the boot partition --- it won't work on buster and bullseye.
"Fortunately", there were were massive number security vulnerabilities
in grub2 which forced a backport of grub2 2.06 to bullseye and buster,
so if you have the security updates enabled, you'll probably be OK ---
but it was only because of massive number of security problems forced
that backport.


In any case, a version of grub that will support the csum_seed feature
will be landing in Bookworm in just a few days.  So at that point,
you'll be able to create VM images for Bookworm and Sid that will work
with the e2fsprogs in sid.  The current plan of record is that it will
only be at that point that e2fsprogs will be allowed to migrate into
Bookworm.

For slowly moving upstreams like grub2, distributions *have* to take
updates before grub2 finally gets around to doing a release --- to get
security fixes if nothing else!  The support for csum_seed has been in
Fedora and other distributions for a while, since the patches had
landed in grub2 in June 2021.  I probably should have made sure the
feature had landed in Debian's grub2 packaging earlier; that's my bad,
and my apologies for that.

Note that Debian's grub2 has well over 100 patches, nearly all of
which are backports from grub2's git repo.  So the argument that
"there doesn't exist a formally released grub2 release" isn't
particularly compelling, since all the distros are backporting
patches.  The only question is how *many* commits release has an
individual distribution taken.


By the way, in the case of the csum_seed feature, it's pretty
straightforward to just run "tune2fs -O ^metadata_csum_seed
/tmp/boot.img".  If the UUID has been changed since the file system
was created, you'll have to do this while the file system is unmounted
and it will take a few seconds, but that's almost certainly not the
case with vmdb2.

	       	      	       	       - Ted

Am Dienstag, dem 14.02.2023 um 12:34 -0500 schrieb Theodore Ts'o:
[..]

As soon as this version hits testing, you have successfully disabled
the last working environment to use vmdb2 to create images of Ubuntu
and Debian. As soon as this version hits Testing, one then can no
longer build images for either any Ubuntu release nor Debian Bullseye
or older. I can then only build images for Bookworm and Sid. Please
think about that.

You *seriously* break the debootstrap method of installing Debian or
Ubuntu. vmdb2 ist just another tool that is broken by now, a tool that
I use very often. I explained the impacts of what you are doing often
enough now. By now the impact should be clear. And you are still not
dealing with the aftermath of your intended action?! You haven't done a
proper transition. You haven't given any affected toolchain the
necessary time to adopt. You haven't documented how to disable that
breaking change when creating filesystems for distributions where grub
does not support this (there is not even a NEWS entry). You haven't
even announced that breaking change. And yet you are going to break one
of our two standard methods of installing Debian or Ubuntu. How is any
of what you have been saying so far addressing any of these issues?

Well, how do you intend to distribute that information, so people who
have to use the deboostrap method to install a Debian or Ubuntu release
with a grub not supporting csum_seed (basically all existing releases,
except for the upcoming Bookworm) know that?

I do not understand why you are pushing 1.47.0 over 1.46.6, which you
had uploaded just five days before the former. You haven't even
presented a reason yet.

Regards, Daniel

The number of users who use vmdb2 are quite small; for those users who
do, there is a simple fix.  You can either diable the feature using
tune2fs, or you can just make an edit to your /etc/mke2fs.conf file to
not enable the feature.  A one line change to /etc/mke2fs.conf is
hardly what I'd call "impossible".

As you have pointed out, this is not a debootstrap issue, since it
doesn't create the file system.  The uestion is in how the file system
is created, and this is not hard to fix.  You can just run "mke2fs -O
^metadata_csum_seed _file_or_block_device_"; you can run "tune2fs -O
^metadata_csum_seed _file_or_block_device"; you can make a one-line
change to /etc/mke2fs.conf.

Sorry, as far as I'm concerned, vmdb2 is not that important.  We don't
document in a NEWS entry or anywhere else, how to build a binary that
links against a newer version of glibc so it will work on an older
system.  And I would consider the compiler toolchain to be far more
common a usecase than vmdb2.  Indeed, your use of "toolchain" for file
system utilities.... is a new one for me.  I've never heard the term
"toolchain" used in such a way before.

It has anumber of new features that will improve ext4's performance on
highly parallel workloads; it makes it possible for cloud VM's to be
able to safely update the root file systems's UUID while it is
mounted, among other changes.

					- Ted

Am Dienstag, dem 14.02.2023 um 16:53 -0500 schrieb Theodore Ts'o:

[snip]

Your arrogant and ignorant attitude is frustrating, to say the least.
You don't care about the mess you create, for a feature, that probably
only a handful of people will ever need (I did a quick search and
didn't find anything regarding this feature - so why turn it on by
default then?). But yet you have to make it the default and break
running toolchains and methods. Talking to you is pointless. You cost
me hours of useless work yesterday and today because you ignore the
rules we have set out as a project to not frustrate each other.

I have reported this to the release team now.

Daniel

Control: clone -1 -2
Control: reassign -2 vmdb2 0.26-2

Regardless of the outcome of #1031325, this issue will need to be fixed
in vmdb2 eventually. vmdb2, similar to other bootstraping tools, has to
account for the feature and disable it if necessary for older
distributions.

Cloning and reassign to vmdb2.

Cheers

Control: clone -1 -2
Control: reassign -2 fai 6.0
Control: clone -1 -3
Control: reassign -3 grml-debootstrap 0.102

Based on more feedback from #10313225, I am also cloning and reassigning
this issue to fai and grml-debootstrap. Dear maintainers, please check
whether this issue is relevant for your packages.

Cheers

Control: clone -1 -2
Control: reassign -2 freedom-make 0.32
Control: clone -1 -3
Control: reassign -3 debos 1.1.1-2

The same appears to apply for debos and freedom-maker. Dear maintainers,
please check whether this issue is relevant for your packages.

Paul

It turns out for debos the situation is a bit different. Since debos
uses packages on the host to prepare the partitions, new features in
e2fsprogs from unstable which are unsupported in the target suite
causes the built system to not boot.

One such failure at boot-time of a Debian testing system is when
systemd-fsck tries to check a filesystem, it fails to mount the device:

  /dev/mmcblk0p5 has unsupported feature(s): FEATURE_C12

So, in short we cannot run images built for suites with older e2fsprogs
on a host system which has e2fsprogs >=1.47.0-1.

Fixing grub2 will not fix the issue for debos, so I have retitled the
bug.

In debos with newer e2fsprogs, we can set the following flag on a
partition to disable the feature which allows the system running older
e2fsprogs to boot, but older e2fsprogs doesn't allow us to set this
flag:

  features: [ "^orphan_file" ]

So to my mind, the bug is in e2fsprogs as it doesn't maintain backwards
compatibility.

I have reduced the severity of the bug since a workaround for debos
does exist. I wonder whether the bug should be reassigned to e2fsprogs
?

E2fsprogs is always going to be adding new features.  Even if I dumb
down /etc/mk2fs.conf to not enable this feature in Bookworm, it *will*
enable this new feature in the next release of Debian.  This is true
for all file systems; we add new features to improve user experience.

So packages that are creating file systems to be consumed by older
target OS's must either (a) be aware of these changes for all file
systems (for example XFS is enabling the "bigtime" feature by default
in Bookworm to fix y2038 bug by expanding the size of the timestamp in
their inodes), OR, (b) run the mke2fs from the target OS in build
chroot.

This might get tricky for some target OS's, such as for example, GNU
Hurd, but the Hurd *absolutely* needs to either use mkfs.ext4 from
Hurd, or if you are using the mke2fs from Linux, you'll need
"mkfs.ext2 -O hurd".

The assumption that mkfs's from file systems never change and provide
full backwards compatibility for all target OS's is very much an
anti-pattern.

Best regards,

						- Ted

Seems like the bookworm / metadata_csum_seed problem has happened in the
past for a different e2fsprogs feature and is discussed in the above
upstream github issue, so I'm marking this as forwarded to have the
two issues linked.

Regards,
Andreas Henriksson