Dear maintainer(s) of debian-installer,

I've been working on reproducible live ISO images for some time now [1] and the
images can be generated in a reproducible manner.
As the next step, I want to test the functionality of the live ISO images and
recently I started working with Philip Hands to get them being tested in
openQA.

I have noticed that the officially released version debian-installer [2][3]
will not work for bookworm and sid, because the kernel version in the debian-
installer does not match the current kernel version. You recently fixed this in
git [4].

Could you release a new version of debian-installer for bookworm and sid?
Or do you recommend a different (release) strategy?
I'm aware of the daily images [5], but they are currently not being
snapshotted, which makes it impossible to reproduce an image after the older
images have been removed from [5].

With kind regards,
Roland Clobus

[1] https://wiki.debian.org/ReproducibleInstalls/LiveImages
[2]
https://snapshot.debian.org/archive/debian/20220305T025031Z/dists/sid/main/installer-
amd64/20210731/
[3] https://deb.debian.org/debian/dists/sid/main/installer-amd64/current/
[4] https://salsa.debian.org/installer-team/debian-
installer/-/blob/f810235e642e7ed266cc6a41b8fccd864180714a
[5] https://d-i.debian.org/daily-images/amd64/daily/

Hi,

Roland Clobus <rclobus@rclobus.nl> (2022-03-05):

We could, and should, release a new d-i and possibly an Alpha 1 at some
point, but I don't have a specific timeline for that.

A new debian-installer upload (prelude to the aforementioned Alpha 1) is
only going to help until src:linux gets a new ABI bump, so that's only
going to be temporary anyway.

If you're using a specific build, you could mirror it on your side, and
then have a way to point at the mirrored copy so that you wouldn't
depend on d-i.d.o's contents (that's an approach seen in various
projects, e.g.  time-based snapshots and tagged snapshots in Tails, even
if that's for Debian as a whole, not d-i)?

How long do you need to go back / how long do you need to keep a given
build? Maybe we could just keep (some) builds for a longer while there,
but that's at 90 days already.


Cheers,

+mailing list rb-general

Hi,

Understood. I assume that an Alpha 1 release will be made somewhere near
the release date of bookworm.

Indeed. A new debian-installer upload would need to happen in lock-step
with every new ABI in src:linux, to guarantee a consistent state of d-i.
This could mean quite some work on your side.

I do not know how much work it is to release a new version of
debian-installer. Currently the state of the official repository
(deb.debian.org) is a non-working installer for bookworm and sid.

I'm looking at possible solutions here (that's why I've added the
rb-general mailing list):
* (Manually) do official releases of debian-installer more often
   (as I wrote, openQA will soon have some tests that detect when the
kernel version got out-of-sync)
* Automatically release git snapshots to deb.d.o instead of d-i.d.o
* Extend snapshot.d.o and/or snapshot.notset.fr to cover d-i.d.o in
addition to deb.d.o
* No changes, and accept that older images cannot be recreated (this
option is not preferred by me)
* Other ...

Looking at https://d-i.debian.org/daily-images/amd64/, the current
history I can see is about 15 days.

While investigating reproducible issues I personally tend to pick some
timestamp and work on that for a longer period of time. 90 days would
suffice completely for my purpose.

With kind regards,
Roland Clobus

Roland Clobus <rclobus@rclobus.nl> (2022-03-05):

In the past I've tried to have an Alpha 1 released after a few months
into the new release cycle, then aim for something like a release every
1-2 months.

But the archive can disagree from time to time, and lately, I'm rather
busy with other things…

Uploading more often could be doable; the current approach has been to
upload whenever we were getting close to wanting a new release of the
installer, following up with 1-2 more uploads if things didn't work out
immediately.

Doable, as stated above.

Well, I do follow kernel uploads and ABI bumps closely, so that's a
problem that's already solved (git says since 2013).

FWIW we're seeing mutiple hours to multiple days of delay for some
architectures (we need the *-signed packages to get processed).

Uploading untested stuff to the archive, automatically? No, thanks.

Maybe avoidable if we go for more frequent uploads.

Right, ISTR that was around 30 days, glanced at the graph before hitting
the sack and didn't check thoroughly: we indeed clean more frequently
than showed on the graph.

At the moment, daily-images is 42G, 6.8G of it being amd64. If we
estimate a ×6 in size for just that architecture, that's nearly doubling
that size, and it's only covering that one architecture. The machine is
shared across various services, but has 121G free at the moment, so we
could probably try something like that (maybe some middleground like 45
days for starters) if that helped in the very short term? Or would you
need to look at more architectures immediately?


Cheers,

Hello Cyril,

Understood. So I've focussed on building the daily image myself, using
the git version.
This will
1) allow me to generate installer snapshots for testing and unstable
that have their correct kernel version (because you diligently fix that
in the git repo)
2) save you the time of doing new releases for testing and unstable.

After some hick-ups, I've got a working version now.
(See my aborted MR27:
https://salsa.debian.org/installer-team/debian-installer/-/merge_requests/27)

I'm currently preparing a 'rebuild' script for live-build, that will
reproducibly re-generate an image, including the installer that matches
that specific point in time.

If you are interested, it might be usable for the daily-build script as
well, because it will not use the timestamp for SOURCE_DATE_EPOCH of
midnight (the time the script is started), but the timestamp of the last
completed snapshot of the archive.

I can go back as far as I want right now. There is no need any more for
the d-i.debian.org snapshots when I recreate the installer.

With kind regards,
Roland