I am building Debian Live images using mmdebstrap.
I noticed they were not reproducible.
I eventually narrowed it down to dbus:
bash5$ for i in 1 2; do SOURCE_DATE_EPOCH=1 mmdebstrap bullseye $i.squashfs --logfile=$i.log || echo DERP; done && cksum 1.squashfs 2.squashfs
2581161214 57483264 1.squashfs
2581161214 57483264 2.squashfs
bash5$ for i in 1 2; do SOURCE_DATE_EPOCH=1 mmdebstrap bullseye $i.squashfs --logfile=$i.log --include=dbus || echo DERP; done && cksum 1.squashfs 2.squashfs
1368751007 57929728 1.squashfs
2280740337 57929728 2.squashfs
bash5$ mmdebstrap --version
mmdebstrap 0.7.5
After some investigation, I discovered that inside the image (1.squashfs), /etc/machine-id is absent, but /var/lib/dbus/machine-id is present.
An unreleased version of mmdebstrap changes sets /etc/machine-id to "uninitialized".
https://gitlab.mister-muffin.de/josch/mmdebstrap/commit/3c37d692a0d0bbc829b0832ad20ff472c74a7a62
This did NOT fix the problem:
bash5$ git pull
Already up to date.
bash5$ git describe --always --tags
0.7.5-50-g7d472ca
bash5$ export PATH=$PWD:$PATH
bash5$ which mmdebstrap
/home/twb/Desktop/mmdebstrap/mmdebstrap
bash5$ for i in 1 2; do SOURCE_DATE_EPOCH=1 mmdebstrap bullseye /tmp/$i.squashfs --verbose --logfile=/tmp/$i.log --include=dbus || echo DERP; done && cksum /tmp/[12].squashfs
3289132048 57929728 /tmp/1.squashfs
449302736 57929728 /tmp/2.squashfs
bash5$ sudo mkdir /tmp/1 /tmp/2
bash5$ sudo mount /tmp/1.squashfs /tmp/1
bash5$ sudo mount /tmp/2.squashfs /tmp/2
bash5$ sudo mount -t tmpfs none /tmp/1/dev
bash5$ sudo mount -t tmpfs none /tmp/2/dev
bash5$ sudo GIT_PAGER= git diff --no-index /tmp/1 /tmp/2
diff --git a/tmp/1/var/lib/dbus/machine-id b/tmp/2/var/lib/dbus/machine-id
index 3edc593..ec187b7 100644
--- a/tmp/1/var/lib/dbus/machine-id
+++ b/tmp/2/var/lib/dbus/machine-id
@@ -1 +1 @@
-0b51d6ce222c427f9cf10a49e2db92a7
+e003fd9d3ca240d786eda2c574f7b791
bash5$ cat /tmp/1/etc/machine-id
uninitializedbash5$ cat /tmp/2/etc/machine-id
uninitializedbash5$ sudo umount /tmp/1/dev /tmp/2/dev /tmp/1 /tmp/2
I also had a guess trying to run "dbus-uuiden --ensure", but it did not help:
bash5$ mmdebstrap bullseye /tmp/1.squashfs --include=dbus --customize-hook='grep ^ $1/etc/machine-id $1/var/lib/dbus/machine-id; printf uninitalized >$1/etc/machine-id; printf uninitalized >$1/etc/machine-id $1/var/lib/dbus/machine-id; chroot $1 dbus-uuidgen --ensure; ls -lids $1/etc/machine-id $1/var/lib/dbus/machine-id; grep ^ $1/etc/machine-id $1/var/lib/dbus/machine-id; false'
I: automatically chosen mode: unshare
I: chroot architecture amd64 is equal to the host's architecture
I: automatically chosen format: squashfs
I: using /tmp/mmdebstrap.CxI5RYbsZJ as tempdir
W: tar2sqfs does not support extended attributes from the 'system' namespace
I: running apt-get update...
done
I: downloading packages with apt...
done
I: extracting archives...
done
I: installing essential packages...
done
I: installing remaining packages inside the chroot...
done
done
I: running --customize-hook in shell: sh -c 'grep ^ $1/etc/machine-id $1/var/lib/dbus/machine-id; printf uninitalized >$1/etc/machine-id; printf uninitalized >$1/etc/machine-id $1/var/lib/dbus/machine-id; chroot $1 dbus-uuidgen --ensure; ls -lids $1/etc/machine-id $1/var/lib/dbus/machine-id; grep ^ $1/etc/machine-id $1/var/lib/dbus/machine-id; false' exec /tmp/mmdebstrap.CxI5RYbsZJ
/tmp/mmdebstrap.CxI5RYbsZJ/etc/machine-id:cb3779e3f4e64aebab438a1dee355b59
/tmp/mmdebstrap.CxI5RYbsZJ/var/lib/dbus/machine-id:cb3779e3f4e64aebab438a1dee355b59
4112969 4 -r--r--r-- 1 root root 12 2021-09-12 02:39 /tmp/mmdebstrap.CxI5RYbsZJ/etc/machine-id
4117680 4 -rw-r--r-- 1 root root 33 2021-09-12 02:39 /tmp/mmdebstrap.CxI5RYbsZJ/var/lib/dbus/machine-id
/tmp/mmdebstrap.CxI5RYbsZJ/etc/machine-id:uninitalized
/tmp/mmdebstrap.CxI5RYbsZJ/var/lib/dbus/machine-id:cb3779e3f4e64aebab438a1dee355b59
E: run_chroot failed: E: command failed: grep ^ $1/etc/machine-id $1/var/lib/dbus/machine-id; printf uninitalized >$1/etc/machine-id; printf uninitalized >$1/etc/machine-id $1/var/lib/dbus/machine-id; chroot $1 dbus-uuidgen --ensure; ls -lids $1/etc/machine-id $1/var/lib/dbus/machine-id; grep ^ $1/etc/machine-id $1/var/lib/dbus/machine-id; false
W: listening on child socket failed:
reading tar header: short read
W: creating tarball failed: E: failed to run tar2sqfs --quiet --no-skip --force --exportable --compressor xz --block-size 1048576 /tmp/1.squashfs
I: removing tempdir /tmp/mmdebstrap.CxI5RYbsZJ...
bash5$ cat /etc/machine-id
029d2e3fb4ee4d5eaa67c315db3ba66d
Let's just check this issue affects sid (latest dbus in Debian)...
bash5$ mmdebstrap sid /tmp/1.squashfs --include=dbus --customize-hook='grep ^ $1/etc/machine-id $1/var/lib/dbus/machine-id; printf uninitalized >$1/etc/machine-id; printf uninitalized >$1/etc/machine-id $1/var/lib/dbus/machine-id; chroot $1 dbus-uuidgen --ensure; ls -lids $1/etc/machine-id $1/var/lib/dbus/machine-id; grep ^ $1/etc/machine-id $1/var/lib/dbus/machine-id; false'
I: automatically chosen mode: unshare
I: chroot architecture amd64 is equal to the host's architecture
I: automatically chosen format: squashfs
I: using /tmp/mmdebstrap.ZLY1AQ7Gjd as tempdir
W: tar2sqfs does not support extended attributes from the 'system' namespace
I: running apt-get update...
done
I: downloading packages with apt...
done
I: extracting archives...
done
I: installing essential packages...
done
I: installing remaining packages inside the chroot...
done
done
I: running --customize-hook in shell: sh -c 'grep ^ $1/etc/machine-id $1/var/lib/dbus/machine-id; printf uninitalized >$1/etc/machine-id; printf uninitalized >$1/etc/machine-id $1/var/lib/dbus/machine-id; chroot $1 dbus-uuidgen --ensure; ls -lids $1/etc/machine-id $1/var/lib/dbus/machine-id; grep ^ $1/etc/machine-id $1/var/lib/dbus/machine-id; false' exec /tmp/mmdebstrap.ZLY1AQ7Gjd
/tmp/mmdebstrap.ZLY1AQ7Gjd/etc/machine-id:9a85960cbe9a4be78f6359211a35cddb
/tmp/mmdebstrap.ZLY1AQ7Gjd/var/lib/dbus/machine-id:9a85960cbe9a4be78f6359211a35cddb
4128767 4 -r--r--r-- 1 root root 12 2021-09-12 02:44 /tmp/mmdebstrap.ZLY1AQ7Gjd/etc/machine-id
4133499 4 -rw-r--r-- 1 root root 33 2021-09-12 02:44 /tmp/mmdebstrap.ZLY1AQ7Gjd/var/lib/dbus/machine-id
/tmp/mmdebstrap.ZLY1AQ7Gjd/etc/machine-id:uninitalized
/tmp/mmdebstrap.ZLY1AQ7Gjd/var/lib/dbus/machine-id:9a85960cbe9a4be78f6359211a35cddb
E: run_chroot failed: E: command failed: grep ^ $1/etc/machine-id $1/var/lib/dbus/machine-id; printf uninitalized >$1/etc/machine-id; printf uninitalized >$1/etc/machine-id $1/var/lib/dbus/machine-id; chroot $1 dbus-uuidgen --ensure; ls -lids $1/etc/machine-id $1/var/lib/dbus/machine-id; grep ^ $1/etc/machine-id $1/var/lib/dbus/machine-id; false
W: listening on child socket failed:
reading tar header: short read
W: creating tarball failed: E: failed to run tar2sqfs --quiet --no-skip --force --exportable --compressor xz --block-size 1048576 /tmp/1.squashfs
I: removing tempdir /tmp/mmdebstrap.ZLY1AQ7Gjd...
I am not sure how to fix this.
The references to machine-id in the dbus sources confused me.
It seems like sometimes it's a link, sometimes it's a symlink, sometimes it's a copy.
AFAICT there's no mention of dbus's machine-id supporting "uninitialized".
Retitling because this does not affect reproducibility *of packages*
(as recommended in Policy §4.15), only reproducibility of whole systems
(chroot/container/image).
The problem here is that traditionally, merely installing the dbus
package - without requiring a reboot or a specific init system - has
been sufficient to get a fully-working D-Bus installation. One of the
properties provided by a fully-working D-Bus installation is that there is
a machine ID (in particular, dbus-launch(1) in the dbus-x11 package
will not work otherwise, but in general, the authors of dbus consider a
missing machine ID to be an incorrect and unsupported installation).
The system bus starts as uid 0 and is able to set up a machine ID for
itself, but the session bus and arbitrary user-defined buses (such
as the one used for AT-SPI) are unprivileged and cannot generate a
machine ID, so they have to rely on something "larger" (like systemd,
or /etc/init.d/dbus, or dbus.postinst) to do that setup.
If I was designing a message-bus system today, I wouldn't include a
machine ID in it; but I don't get to choose the API guarantees that
I've inherited from the original designers of D-Bus, and backwards
compatibility is important to me. I can see why the machine ID was
included, because it's there as a machine-oriented replacement for
the hostname, which has two properties that make it undesirable: it's
non-unique (lots of machines think their name is "debian" or "ubuntu" or
"localhost"), breaking the desirable property that same hostname implies
same machine; and it's human-meaningful, which means sysadmins sometimes
want to change it for cosmetic or administrative reasons, breaking the
desirable property that different hostname means different machine.
Back when D-Bus was designed, NFS-shared home directories and remote
X11 were considered to be essential-to-support, such that D-Bus would
not have been adopted if it did not cope with those; but that means it
needs a reliable way to identify machines among the multiple that can
be sharing a home directory or an X11 display (and no, the hostname is
not enough, for the reasons I mentioned above). Those use-cases are a lot
less important now, and could perhaps even be considered to be deprecated,
but the feature that was necessary to support them remains.
/etc/machine-id is a generalization of the D-Bus machine ID, originating
in systemd. There would be nothing to stop non-systemd machines from
implementing it, but there is a tendency for people who dislike systemd
to reject anything that came from systemd and work against its wider
adoption unless there is absolutely no alternative, so it is not
considered mandatory for Debian systems in general.
As a result, /etc/machine-id is not guaranteed to exist unless/until
the system has been booted successfully with systemd. If the system is
to be used as a "plain" chroot, or a container that will be run without
a full init system (as is conventional with Docker), or a machine that
will boot with sysvinit or some other non-systemd init system, then
there will usually be no /etc/machine-id.
*If* the system is always going to be booted with systemd (and
in particular for systemd-based live-images), then it's safe for
/var/lib/dbus/machine-id to be deleted or replaced with a symlink to
/etc/machine-id; but the dbus package's postinst cannot know whether
this is the case. Even if systemd-sysv happens to be installed already,
that's no guarantee that the system will not be used as a chroot with
no real init system, in which case systemd will be present but dormant,
and nothing will create /etc/machine-id.
If /var/lib/dbus/machine-id is deleted, on systems that boot with systemd,
the tmpfiles snippet /usr/lib/tmpfiles.d/dbus.conf will replace it with a
symlink to /etc/machine-id; or on systems that boot with sysvinit,
a call to dbus-uuidgen in /etc/init.d/dbus will regenerate it. However,
this will not generally happen on non-systemd machines.
When /var/lib/dbus/machine-id is generated by dbus-uuidgen in the dbus
postinst or in /etc/init.d/dbus, if /etc/machine-id exists, dbus-uuidgen
will copy it. In this case it is a copy, not a symlink, because dbus
cannot guarantee that the file /etc/machine-id (which is not conceptually
"owned" by dbus) will not get deleted out from under us.
dbus could in principle create /etc/machine-id instead of
/var/lib/dbus/machine-id, and make /var/lib/dbus/machine-id a symlink to
it, but, again, dbus does not conceptually "own" /etc/machine-id, so this
would create a risk that /etc/machine-id will be deleted by some other
component, breaking the guarantees that dbus aims to provide.
If you want mmdebstrap to generate reproducible images, then I think
the best analogue to "echo uninitialized > /etc/machine-id" would be
to delete /var/lib/dbus/machine-id, allowing it to be re-created during
next boot. However, if there is no such thing as the "next boot" because
the tree being bootstrapped is a chroot or a non-init-system container,
that will result in an incomplete and partially non-functional D-Bus
installation. Knowing whether this is an acceptable tradeoff requires more
context than either dbus.postinst or mmdebstrap has available to them.
I think the best solution to this might be to make /etc/machine-id
part of the "specification" for what makes a Debian system, as an
init-system-independent "API", similar to how we handle /run and
/usr/lib/os-release, but I suspect that people who dislike systemd would
oppose that as a point of principle, and I have higher priorities for how
I want to spend the necessary emotional energy to make contentious things
happen in Debian.
That's because it doesn't. The special keyword "uninitialized" is a
recently-added feature of systemd's handling of /etc/machine-id, which
is newer than the D-Bus machine ID.
*If* the system is booted with systemd, then /etc/machine-id will
be replaced with a real machine ID during early boot, before the
system has booted up far enough for D-Bus components to be running, so
"uninitialized" will never be observable by D-Bus components. However,
if the system is booted with a different init system, or if it is a
chroot/container that is never "booted" at all, then that will not happen.
smcv