#964248 base-installer: Pass "--extra-suites=unreleased" to debootstrap on ports arches

Package:
base-installer
Source:
base-installer
Submitter:
наб
Date:
2021-02-20 00:36:03 UTC
Severity:
normal
#964248#5
Date:
2020-07-04 11:32:35 UTC
From:
To:
Hi!

I've posted these patches to debian-boot@ last month,
but thought I'd try again.

Confer commit messages for detailed references, but the short of it is:
  the base-installer patch pulls in ports arch detexion logic from
  apt-setup and adds the unreleased suite in debootstrap, fixing Hurd
  installs from, i.a., mini ISOs, which require it for a working system,
and
  the apt-setup patch switches to the detection logic
  moved to base-installer.

Thread on debian-hurd@:
<https://lists.debian.org/debian-hurd/2020/06/msg00015.html>

These patches apply cleanly on top of the Salsa HEADs as of today
(e70510edbb6772e0eb98537c3a7175b977ce0423 for base-installer and
 30f36f7e8617f52cbc7fc9942cfbf326d2b773c2 for apt-setup).

Best,
наб
#964248#10
Date:
2020-07-04 17:15:25 UTC
From:
To:
Hello,

наб, le sam. 04 juil. 2020 13:32:35 +0200, a ecrit:

Thanks for the ping. Since there was no concern raised in the meanwhile,
I have pushed the change to the gits.

Samuel
#964248#17
Date:
2020-07-12 11:48:37 UTC
From:
To:
We believe that the bug you reported is fixed in the latest version of
base-installer, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 964248@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Holger Wansing <hwansing@mailbox.org> (supplier of updated base-installer package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)
Format: 1.8
Date: Sun, 12 Jul 2020 13:30:23 +0200
Source: base-installer
Architecture: source
Version: 1.195
Distribution: unstable
Urgency: medium
Maintainer: Debian Install System Team <debian-boot@lists.debian.org>
Changed-By: Holger Wansing <hwansing@mailbox.org>
Closes: 964248
Changes:
 base-installer (1.195) unstable; urgency=medium
 .
   * Team upload
 .
   [ наб ]
   * Pass "--extra-suites=unreleased" to debootstrap for ports architectures,
     fixing, e.g., hurd installs from mini ISOs (Closes: Bug#964248).
 .
   [ Updated translations ]
   * Icelandic (is.po) by Sveinn í Felli
   * Norwegian Bokmal (nb.po) by Allan Nordhøy
Checksums-Sha1:
 5ac36e1e5c5ab3e09babe37e322c432748e993c3 1942 base-installer_1.195.dsc
 8f29ca4407be8e8ea69e1f03bd3e97267507d11a 256916 base-installer_1.195.tar.xz
 1d600980ab9b04bc19c04fea29f487c1513d5f35 5958 base-installer_1.195_amd64.buildinfo
Checksums-Sha256:
 8484ac868579280007ba7cb1051bfd7ebb4f1bf2db09afc5a9b1e2dfe32d3147 1942 base-installer_1.195.dsc
 e2fe77a4e38bb1a4d5bd490053b89f5917f5f5b6f44029ddce3a7177ce586d13 256916 base-installer_1.195.tar.xz
 2350d0653724a3483244bf75e77bff18f4f67ce4cd08fe5898cf3438938acd4e 5958 base-installer_1.195_amd64.buildinfo
Files:
 2bca61a4adc0872cb9c1ccdd22af7e51 1942 debian-installer required base-installer_1.195.dsc
 6f19aea29486c8f9ef5a06a0caf6b46f 256916 debian-installer required base-installer_1.195.tar.xz
 58f6d31fde09f66a7ff1f285f365d96c 5958 debian-installer required base-installer_1.195_amd64.buildinfo
-----BEGIN PGP SIGNATURE-----

iQJJBAEBCgAzFiEESWrG6BRCSzSFCDUpWfGHyhVusHYFAl8K9fQVHGh3YW5zaW5n
QG1haWxib3gub3JnAAoJEFnxh8oVbrB2orcP/0A5vXk6C3xGV4vbL4oBiCNH9LvQ
jJOKTHty8ZbKC8S62te24fI7+oOFM3k7RaPE5Ltp1HMEReaQlrFbbiDTpGpblVKS
J2ugbrb9/oUaNMIf7XRTzHxJVFQthqRHX8mFCs/ggxVu1bOQkJOAoVksXAthOFKY
HYycDPsKpabI8nTwUosIpynq2UDHbjdKMcc1mLTpgD41gZOdg7ZI3psopXjsFE85
lWMRcZ0kw2uPJGLgFOpKHTYmAA0Ra/5H2W/OqrTvF7O1Cv9LClUJXS3Pbf7PcalP
Yja88XCG36dOyXGRp0/4gj8psjVzr9Cc2K7tW/4kYutGWOHFrOO6Br/hLeks7XQQ
m6vhOqTKEIBqkibNk+BMgOb5yEEy+CXvySnR2OWzjhHVYXrrIWSWS8daOvz6inca
l71ReSdCLyVNP+D+9dHeLHqX431deBr1MVAD4jSlMjUSiJOk8jkXEp3+N6vGZYxT
zWXXCHPxi4iTMWzEbTcVJyRcGVePdkVUZxMfyLE7QlypaueXLKjvUrRPqcNTTOst
7YCVKed0dAoa1IOe/mIqdxell9R9xvVfL6eLcQewOSwCtfJsHPQ+dvGTrfDpRO+b
TiHUikZ/3V/Vyg9vxZm0o40ZyWT61Uc9xSCryrnP9rYRY7VfSqR0ihvfEB1wRxeZ
+9vkUzsLV960MLKN
=ylrg
-----END PGP SIGNATURE-----
#964248#22
Date:
2020-07-16 12:59:37 UTC
From:
To:
Thanks, Samuel!

I tried the mini ISO and it worked as expected,
downloading packages from unreleased w/o problems.

It did, however, encounter problems with trying to download
libhogweed and libnettle7 3.5.1+really3.5.1-2+b1,
though the only version I saw in the pool was 3.6-2,
but that's, presumably, unrelated.

наб
#964248#27
Date:
2020-07-28 08:39:40 UTC
From:
To:
Hi,

Thank you for your work to improve the installability of port architectures !

Sorry for jumping so late into the discussion of the patch of bug #964248.

I wonder if the versioned dependency of apt-mirror-setting on base-installer
is necessary. Bug #879130 was solved (thank you !), and since then we could
just write :

is_ports_architecture=$(cat /usr/lib/choose-mirror/port_architecture)

Is it sufficient ?

And we were advised against hard-coding a list of port architectures [1].

I hope it will help nevertheless !

Regards,
JH Chatenet

[1] : https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=690210#31
#964248#32
Date:
2020-07-30 21:00:26 UTC
From:
To:
jhcha54008, le mar. 28 juil. 2020 10:39:40 +0200, a ecrit:

AIUI it is for getting the is_ports_architecture function from
/usr/lib/base-installer/library.sh?

Mmm, possibly, are we sure that choose-mirror is always installed in
d-i? It seems that monolithic/netboot have it anyway, and cdrom installs
it already? наб, perhaps you can try and check that it works?

Yep, sure, better not duplicate the list.

Samuel
#964248#37
Date:
2020-07-30 21:17:33 UTC
From:
To:
Hello,

jhcha54008, le mar. 28 juil. 2020 12:44:42 +0200, a ecrit:

Yes, I know, but it means that something has to merge the packages etc.
ATM I'm doing it through gross scripts for building hurd CD images, but
that's not really sustainable.

It is when we do not have several versions of the same package, which is
the case for various packages for various ports.

Yes, my idea was to keep things the same way, so installations from
netboot, netinst, CD, local mirror would work the same, and we avoid
seeing installability issues in one of the cases and not the others.

Samuel
#964248#42
Date:
2020-07-31 10:50:45 UTC
From:
To:
It was, yes.
I repacked today's hurd netinst with an edited apt-mirror-setup and
/usr/lib/choose-mirror/port_architecture was present with "1" in it
when the script ran, I hope that's a good test?

наб
#964248#47
Date:
2020-07-31 11:15:58 UTC
From:
To:
наб, le ven. 31 juil. 2020 12:50:45 +0200, a ecrit:

Netinst is not the only installation media, you also have monolithic,
netboot, hd-image, cd, and anything I don't remember.

Samuel
#964248#52
Date:
2020-08-04 15:19:44 UTC
From:
To:
Hi,

Here is a patch proposal for this bug report. It aims :
- to be compatible with the install CD produced by debian-cd
- to avoid hard-coding a list of architectures in the Debian-Ports repository
- to make the use of the 'unreleased' distribution switchable.

I hope it will help to find a suitable solution !

Regards,
JH Chatenet


# This is a proposal to reopen bug #964248
# Running debootstrap on architectures from the debian-ports repository
# requires passing an option --extra-suites=unreleased
# (if installing from a debian-ports mirror)
#
# Experienced users may want to ignore the 'unreleased' distribution
# in some circumstances (e.g. installing from a custom mirror without
# 'unreleased'). This option is implemented here via a new debconf variable.
# To activate the option (and ignore 'unreleased') the user can either
# preseed the template 'base-installer/no_unreleased' to true or set
# the option at the boot prompt.
# The option is only intended for experienced users and is therefore not
# offered in a dialog. It can be either preseeded or set at the boot prompt.

diff -Naur a/base-installer-1.195/debian/bootstrap-base.postinst b/base-installer-1.195/debian/bootstrap-base.postinst
--- a/base-installer-1.195/debian/bootstrap-base.postinst	2020-07-05 21:34:10.000000000 +0200
+++ b/base-installer-1.195/debian/bootstrap-base.postinst	2020-08-04 12:25:06.527009301 +0200
@@ -96,8 +96,26 @@
 	if [ "${EXCLUDES}" ]; then
 		exclude="--exclude=${EXCLUDES}"
 	fi
-	if is_ports_architecture "${ARCH}"; then
-		extra_suites="--extra-suites=unreleased"
+	if [ "$is_ports_architecture" != "1" ] || \
+		[ "$DISTRIBUTION" != "sid" ] && \
+		[ "$DISTRIBUTION" != "unstable" ]; then
+		# This is a released architecture : nothing to do
+		:
+	else
+		# This is an unreleased architecture in the Debian-Ports repository.
+		# 'unstable' is the only installable distribution.
+		# Installation from a debian-ports mirror may require packages from
+		# the 'unreleased' distribution too.
+		# There is no 'unreleased' distribution on install CD.
+		if [ "$PROTOCOL" = http ] || [ "$PROTOCOL" = https ] ||[ "$PROTOCOL" = ftp ]; then
+		# Did the user decline to consider the 'unreleased' distribution ?
+		# (e.g. if installing from a custom mirror without 'unreleased')
+			if ! db_get base-installer/no_unreleased || [ "$RET" != true ]; then
+				extra_suites="--extra-suites=unreleased"
+			else
+				log "As requested the 'unreleased' distribution is skipped"
+			fi
+	        fi
 	fi
 	sigcheck="--no-check-gpg"
 	if [ "$PROTOCOL" = http ] || [ "$PROTOCOL" = ftp ]; then
diff -Naur a/base-installer-1.195/debian/bootstrap-base.templates b/base-installer-1.195/debian/bootstrap-base.templates
--- a/base-installer-1.195/debian/bootstrap-base.templates	2019-11-17 22:17:31.000000000 +0100
+++ b/base-installer-1.195/debian/bootstrap-base.templates	2020-08-04 12:17:13.288662639 +0200
@@ -452,3 +452,10 @@
 Type: string
 Description: for internal use; can be preseeded
  Force use of a specific debootstrap script
+
+Template: base-installer/no_unreleased
+Type: boolean
+Default: false
+Description: for internal use; can be preseeded
+ Do not consider packages from the unreleased distribution.
+
diff -Naur a/base-installer-1.195/library.sh b/base-installer-1.195/library.sh
--- a/base-installer-1.195/library.sh	2020-07-05 21:34:10.000000000 +0200
+++ b/base-installer-1.195/library.sh	2020-08-04 12:22:18.410175655 +0200
@@ -888,13 +888,9 @@
 	rm -f "$KERNEL_LIST" "$KERNEL_LIST.unfiltered"
 }
#964248#55
Date:
2020-08-04 15:19:44 UTC
From:
To:
Hi,

Here is a patch proposal for this bug report. It aims :
- to be compatible with the install CD produced by debian-cd
- to avoid hard-coding a list of architectures in the Debian-Ports repository
- to make the use of the 'unreleased' distribution switchable.

I hope it will help to find a suitable solution !

Regards,
JH Chatenet


# This is a proposal to reopen bug #964248
# Running debootstrap on architectures from the debian-ports repository
# requires passing an option --extra-suites=unreleased
# (if installing from a debian-ports mirror)
#
# Experienced users may want to ignore the 'unreleased' distribution
# in some circumstances (e.g. installing from a custom mirror without
# 'unreleased'). This option is implemented here via a new debconf variable.
# To activate the option (and ignore 'unreleased') the user can either
# preseed the template 'base-installer/no_unreleased' to true or set
# the option at the boot prompt.
# The option is only intended for experienced users and is therefore not
# offered in a dialog. It can be either preseeded or set at the boot prompt.

diff -Naur a/base-installer-1.195/debian/bootstrap-base.postinst b/base-installer-1.195/debian/bootstrap-base.postinst
--- a/base-installer-1.195/debian/bootstrap-base.postinst	2020-07-05 21:34:10.000000000 +0200
+++ b/base-installer-1.195/debian/bootstrap-base.postinst	2020-08-04 12:25:06.527009301 +0200
@@ -96,8 +96,26 @@
 	if [ "${EXCLUDES}" ]; then
 		exclude="--exclude=${EXCLUDES}"
 	fi
-	if is_ports_architecture "${ARCH}"; then
-		extra_suites="--extra-suites=unreleased"
+	if [ "$is_ports_architecture" != "1" ] || \
+		[ "$DISTRIBUTION" != "sid" ] && \
+		[ "$DISTRIBUTION" != "unstable" ]; then
+		# This is a released architecture : nothing to do
+		:
+	else
+		# This is an unreleased architecture in the Debian-Ports repository.
+		# 'unstable' is the only installable distribution.
+		# Installation from a debian-ports mirror may require packages from
+		# the 'unreleased' distribution too.
+		# There is no 'unreleased' distribution on install CD.
+		if [ "$PROTOCOL" = http ] || [ "$PROTOCOL" = https ] ||[ "$PROTOCOL" = ftp ]; then
+		# Did the user decline to consider the 'unreleased' distribution ?
+		# (e.g. if installing from a custom mirror without 'unreleased')
+			if ! db_get base-installer/no_unreleased || [ "$RET" != true ]; then
+				extra_suites="--extra-suites=unreleased"
+			else
+				log "As requested the 'unreleased' distribution is skipped"
+			fi
+	        fi
 	fi
 	sigcheck="--no-check-gpg"
 	if [ "$PROTOCOL" = http ] || [ "$PROTOCOL" = ftp ]; then
diff -Naur a/base-installer-1.195/debian/bootstrap-base.templates b/base-installer-1.195/debian/bootstrap-base.templates
--- a/base-installer-1.195/debian/bootstrap-base.templates	2019-11-17 22:17:31.000000000 +0100
+++ b/base-installer-1.195/debian/bootstrap-base.templates	2020-08-04 12:17:13.288662639 +0200
@@ -452,3 +452,10 @@
 Type: string
 Description: for internal use; can be preseeded
  Force use of a specific debootstrap script
+
+Template: base-installer/no_unreleased
+Type: boolean
+Default: false
+Description: for internal use; can be preseeded
+ Do not consider packages from the unreleased distribution.
+
diff -Naur a/base-installer-1.195/library.sh b/base-installer-1.195/library.sh
--- a/base-installer-1.195/library.sh	2020-07-05 21:34:10.000000000 +0200
+++ b/base-installer-1.195/library.sh	2020-08-04 12:22:18.410175655 +0200
@@ -888,13 +888,9 @@
 	rm -f "$KERNEL_LIST" "$KERNEL_LIST.unfiltered"
 }
#964248#60
Date:
2020-08-04 17:10:06 UTC
From:
To:
Le vendredi 31 juillet à 13h 15mn 58s (+0200), Samuel Thibault a écrit :

Hi again,

As far as I can check, the file :

/usr/lib/choose-mirror/port_architecture

is shipped in the udeb package choose-mirror-bin
and it is present everywhere choose-mirror-bin is
unpacked. choose-mirror depends on choose-mirror-bin
and apt-setup-mirror depends on choose-mirror, so it's
there when apt-setup-mirror runs (actually, is configured).

debian-installer includes choose-mirror in the native-hd,
monolithic, netboot, netboot-apus and generic images [1].

choose-mirror is not included in the cdrom image of
debian-installer, but loaded by anna from the cdrom.
debian-cd includes choose-mirror in the dynamic task
'debian-installer' and in the businesscards [2].

I hope it will help !

Regards,
JH Chatenet

[1] : build/pkg-lists/*/common in source package debian-installer
[2] : data/{bullseye,buster,jessie,stretch}/*_businesscard_udeb_include
in source package debian-cd
#964248#65
Date:
2020-08-16 22:38:15 UTC
From:
To:
Hi!

This seems to have broken debian-installer on Debian Ports.

Installing the base system now fails with:

Aug 16 22:34:35 debootstrap: /usr/sbin/debootstrap --components=main --debian-installer --resolve-deps --include=debian-ports-archive-keyring,debian-ports-archive-keyring --extra-suites=unreleased --no-check-gpg sid /target file:///cdrom/
Aug 16 22:34:40 base-installer: error: exiting on error base-installer/debootstrap-failed
Aug 16 22:34:41 main-menu[235]: WARNING **: Configuring 'bootstrap-base' failed with error code 1
Aug 16 22:34:41 main-menu[235]: WARNING **: Menu item 'bootstrap-base' failed.

Has this actually been tested to work?

Thanks,
Adrian
#964248#70
Date:
2020-08-16 22:49:02 UTC
From:
To:
Hello,

John Paul Adrian Glaubitz, le lun. 17 août 2020 00:38:15 +0200, a ecrit:

AIUI it was only tested on the mini iso.


I noticed the issue, and worked on it, there is:

https://salsa.debian.org/images-team/debian-cd/-/merge_requests/7

and then a couple other changes, I'm waiting for the above to be merged
before I can submit them.

Samuel
#964248#75
Date:
2020-08-16 22:55:08 UTC
From:
To:
Hi!

Well, then it should not have been merged :(.

What happens when a port does not have any packages in "unreleased"? I checked
the patch and it enables "unreleased" unconditionally. I'm not sure whether
that's a good idea.

FWIW, I have commit access to debian-cd, so I can merge your changes but I would
like to discuss first whether enabling "unreleased" unconditionally will work
even if a port doesn't have any package in this suite.

Thanks,
Adrian
#964248#80
Date:
2020-08-17 08:24:59 UTC
From:
To:
Hello!

I'm not sure I understand this way of addressing this issue.

The proper way is obviously to fix the p11-kit package, not hack debian-installer
to include packages from "unreleased" which is not a regular package archive.

And if you need to build custom CD images which include custom packages, you
can just do that with debian-cd [1].

Adrian
#964248#85
Date:
2020-08-17 08:48:39 UTC
From:
To:
John Paul Adrian Glaubitz, le lun. 17 août 2020 10:36:43 +0200, a ecrit:

Then opinions differ and we need to discuss about it among porters
because that's really not something I have seen specific to the Hurd, I
have seen other porters wanting this.

netdde contains the network drivers, so yes, it is pretty much part of
the base system. Strictly speaking it could be installed "later" at the
task installation stage, but people are supposed to be able to skip that
step and still having a system that works.

It does. I'm just here mentioning one of the cases that I have seen
mentioned from other ports. I don't know the details there, but there
are probably others like firmware packages, etc.  Yes probably these
can also be shipped by hand on debian-cd, but that looks to me like
duplicate work while we can simply enable unreleased.

Sure. But that's very convenient to keep things installable without
having to wait for sid to catch up with whatever issue is encountered by
the port. Otherwise the sid upload latency makes the port uninstallable
for long periods of time, possibly all the time when these periods
overlap.

Samuel
#964248#88
Date:
2020-08-17 08:17:24 UTC
From:
To:
Hello Samuel!

OK, that's a possible workaround. However, ...

... I don't quite understand why "unreleased" is supposed to be part of the
base installation. "unreleased" is not a regular package archive and all
packages in there are usually only added temporarily.

Does Debian Hurd require packages from "unreleased" for it's base system?

Since "unreleased" is not properly maintained due to the fact that all
packages there are built and uploaded manually, I rather prefer them not
to be enabled by default, especially not for the base system.

Adrian
#964248#93
Date:
2020-08-17 08:28:54 UTC
From:
To:
John Paul Adrian Glaubitz, le lun. 17 août 2020 10:17:24 +0200, a ecrit:

For ports architectures, it often is, because the sid archive does not
contain various packages that ports need.

AFAIK that's not what ports' daily life really is.

The Hurd needs netdde from there. It happens that the hurd package
itself is still in sid but that's only by luck because there's the
arch:all hurd-doc package.

But that's far from being specific to the Hurd. Various ports have
various requirements such as bootloaders.

They are uploaded manually by the porters themselves. They are
responsible for what they upload there, so I don't see why it would
be harmful? On the contrary, it provides flexibility for fixing
installability without having to wait for sid to have fixes uploaded.

Samuel
#964248#98
Date:
2020-08-16 23:06:05 UTC
From:
To:
John Paul Adrian Glaubitz, le lun. 17 août 2020 00:55:08 +0200, a ecrit:

My debian-cd changes create empty files so that d-i is happy (exactly
like is done for -backports).
list looks dirty, and trying whether the suite is available is probably
quite complex to implement.

Samuel
#964248#103
Date:
2020-08-17 08:36:27 UTC
From:
To:
Samuel Thibault, le lun. 17 août 2020 10:28:54 +0200, a ecrit:

See also

9bb019085356 ('Enable the "unreleased" suite for ports architectures.')
in the apt-setup package.

Samuel
#964248#106
Date:
2020-08-17 08:36:43 UTC
From:
To:
Hello!

Well, that's not correct since the only additional packages needed are some
bootloaders (emile for m68k, for example) and those are shipped on the
installation CD.

In the case of Debian Hurd, the case here is a package temporarily being
sourced from "unreleased" because the version from unstable FTBFS on
unstable.

Yes, it is. I actually think I have a good overview over this.

And that package is required for the base-system and cannot be installed
later?

These are always shipped on the installation CD. It also affects m68k
only as alpha, hppa, powerpc, ppc64, sh4 and sparc64 use either GRUB or have
their bootloaders in unstable.

Does Hurd not use GRUB?

The problem with "unreleased" is that there are no binNMUs and no proper
maintenance by a package maintainer. I always try to avoid using "unreleased"
whenever possible.
#964248#111
Date:
2020-08-17 08:33:40 UTC
From:
To:
John Paul Adrian Glaubitz, le lun. 17 août 2020 10:24:59 +0200, a ecrit:

That wasn't actually the reason that convinced me to commit this. See my
other mail.

That does not work for the mini iso.

Samuel
#964248#116
Date:
2020-08-17 09:05:19 UTC
From:
To:
John Paul Adrian Glaubitz, le lun. 17 août 2020 10:39:48 +0200, a ecrit:

What I'm seeing is that ports struggle to get to work at all. Yes, this
particular commit was not helping at all, completely agree, and we have
to find solutions that fit everybody's needs.

Building a Debian CD is way more difficult than building a mini.iso.

Yes, with debian-cd one can stuff additional packages in the local/
directory, but again that's not so trivial, and that's duplicating
the work of making them available on network archives for the
already-installed systems.

Samuel
#964248#121
Date:
2020-08-17 09:19:34 UTC
From:
To:
I have reverted the change for now.

But we need to discuss among various porters, I have really seen desire
for enabling the unreleased suite in ports other than the hurd port.

Samuel
#964248#134
Date:
2020-08-19 09:30:29 UTC
From:
To:
Hi,

here is a revised version of a patch proposal
with more detailed log outputs.

I hope it will help !

Regards,
JH Chatenet

# This is a proposal to reopen bug #964248
# Running debootstrap on architectures from the debian-ports repository
# requires passing an option --extra-suites=unreleased
# (if installing from a debian-ports mirror)
#
# Experienced users may want to ignore the 'unreleased' distribution
# in some circumstances (e.g. installing from a custom mirror without
# 'unreleased'). This option is implemented here via a new debconf variable.
# To activate the option (and ignore 'unreleased') the user can either
# preseed the template 'base-installer/no_unreleased' to true or set
# the option at the boot prompt.
# The option is only intended for experienced users and is therefore not
# offered in a dialog. It can be either preseeded or set at the boot prompt.

diff -Naur a/base-installer/debian/bootstrap-base.postinst b/base-installer/debian/bootstrap-base.postinst
--- a/base-installer/debian/bootstrap-base.postinst	2020-08-17 11:16:15.000000000 +0200
+++ b/base-installer/debian/bootstrap-base.postinst	2020-08-18 14:50:01.193310459 +0200
@@ -96,6 +96,29 @@
 	if [ "${EXCLUDES}" ]; then
 		exclude="--exclude=${EXCLUDES}"
 	fi
+	if [ "$is_ports_architecture" != "1" ] || \
+		[ "$DISTRIBUTION" != "sid" ] && \
+		[ "$DISTRIBUTION" != "unstable" ]; then
+		# This is a released architecture : nothing to do
+		:
+	else
+		# This is an unreleased architecture in the Debian-Ports repository.
+		log "Debian-Ports architecture detected"
+		# 'unstable' is the only installable distribution.
+		# Installation from a debian-ports mirror may require packages from
+		# the 'unreleased' distribution too.
+		# There is no 'unreleased' distribution on install CD.
+		if [ "$PROTOCOL" = http ] || [ "$PROTOCOL" = https ] ||[ "$PROTOCOL" = ftp ]; then
+		# Did the user decline to consider the 'unreleased' distribution ?
+		# (e.g. if installing from a custom mirror without 'unreleased')
+			if ! db_get base-installer/no_unreleased || [ "$RET" != true ]; then
+				extra_suites="--extra-suites=unreleased"
+				log "Using the 'unreleased' distribution"
+			else
+				log "As requested the 'unreleased' distribution is skipped"
+			fi
+		fi
+	fi
 	sigcheck="--no-check-gpg"
 	if [ "$PROTOCOL" = http ] || [ "$PROTOCOL" = ftp ]; then
 		if type gpgv >/dev/null; then
diff -Naur a/base-installer/debian/bootstrap-base.templates b/base-installer/debian/bootstrap-base.templates
--- a/base-installer/debian/bootstrap-base.templates	2020-07-04 19:10:15.000000000 +0200
+++ b/base-installer/debian/bootstrap-base.templates	2020-08-18 14:26:03.050179093 +0200
@@ -452,3 +452,10 @@
 Type: string
 Description: for internal use; can be preseeded
  Force use of a specific debootstrap script
+
+Template: base-installer/no_unreleased
+Type: boolean
+Default: false
+Description: for internal use; can be preseeded
+ Do not consider packages from the unreleased distribution.
+
diff -Naur a/base-installer/library.sh b/base-installer/library.sh
--- a/base-installer/library.sh	2020-08-17 11:16:15.000000000 +0200
+++ b/base-installer/library.sh	2020-08-18 14:51:34.605773667 +0200
@@ -887,3 +887,10 @@
 cleanup () {
 	rm -f "$KERNEL_LIST" "$KERNEL_LIST.unfiltered"
 }
+
+is_ports_architecture=
+if [ -r /usr/lib/choose-mirror/port_architecture ]; then
+	log "/usr/lib/choose-mirror/port_architecture is present"
+	is_ports_architecture=$(cat /usr/lib/choose-mirror/port_architecture)
+fi
+
#964248#137
Date:
2020-08-19 09:30:29 UTC
From:
To:
Hi,

here is a revised version of a patch proposal
with more detailed log outputs.

I hope it will help !

Regards,
JH Chatenet

# This is a proposal to reopen bug #964248
# Running debootstrap on architectures from the debian-ports repository
# requires passing an option --extra-suites=unreleased
# (if installing from a debian-ports mirror)
#
# Experienced users may want to ignore the 'unreleased' distribution
# in some circumstances (e.g. installing from a custom mirror without
# 'unreleased'). This option is implemented here via a new debconf variable.
# To activate the option (and ignore 'unreleased') the user can either
# preseed the template 'base-installer/no_unreleased' to true or set
# the option at the boot prompt.
# The option is only intended for experienced users and is therefore not
# offered in a dialog. It can be either preseeded or set at the boot prompt.

diff -Naur a/base-installer/debian/bootstrap-base.postinst b/base-installer/debian/bootstrap-base.postinst
--- a/base-installer/debian/bootstrap-base.postinst	2020-08-17 11:16:15.000000000 +0200
+++ b/base-installer/debian/bootstrap-base.postinst	2020-08-18 14:50:01.193310459 +0200
@@ -96,6 +96,29 @@
 	if [ "${EXCLUDES}" ]; then
 		exclude="--exclude=${EXCLUDES}"
 	fi
+	if [ "$is_ports_architecture" != "1" ] || \
+		[ "$DISTRIBUTION" != "sid" ] && \
+		[ "$DISTRIBUTION" != "unstable" ]; then
+		# This is a released architecture : nothing to do
+		:
+	else
+		# This is an unreleased architecture in the Debian-Ports repository.
+		log "Debian-Ports architecture detected"
+		# 'unstable' is the only installable distribution.
+		# Installation from a debian-ports mirror may require packages from
+		# the 'unreleased' distribution too.
+		# There is no 'unreleased' distribution on install CD.
+		if [ "$PROTOCOL" = http ] || [ "$PROTOCOL" = https ] ||[ "$PROTOCOL" = ftp ]; then
+		# Did the user decline to consider the 'unreleased' distribution ?
+		# (e.g. if installing from a custom mirror without 'unreleased')
+			if ! db_get base-installer/no_unreleased || [ "$RET" != true ]; then
+				extra_suites="--extra-suites=unreleased"
+				log "Using the 'unreleased' distribution"
+			else
+				log "As requested the 'unreleased' distribution is skipped"
+			fi
+		fi
+	fi
 	sigcheck="--no-check-gpg"
 	if [ "$PROTOCOL" = http ] || [ "$PROTOCOL" = ftp ]; then
 		if type gpgv >/dev/null; then
diff -Naur a/base-installer/debian/bootstrap-base.templates b/base-installer/debian/bootstrap-base.templates
--- a/base-installer/debian/bootstrap-base.templates	2020-07-04 19:10:15.000000000 +0200
+++ b/base-installer/debian/bootstrap-base.templates	2020-08-18 14:26:03.050179093 +0200
@@ -452,3 +452,10 @@
 Type: string
 Description: for internal use; can be preseeded
  Force use of a specific debootstrap script
+
+Template: base-installer/no_unreleased
+Type: boolean
+Default: false
+Description: for internal use; can be preseeded
+ Do not consider packages from the unreleased distribution.
+
diff -Naur a/base-installer/library.sh b/base-installer/library.sh
--- a/base-installer/library.sh	2020-08-17 11:16:15.000000000 +0200
+++ b/base-installer/library.sh	2020-08-18 14:51:34.605773667 +0200
@@ -887,3 +887,10 @@
 cleanup () {
 	rm -f "$KERNEL_LIST" "$KERNEL_LIST.unfiltered"
 }
+
+is_ports_architecture=
+if [ -r /usr/lib/choose-mirror/port_architecture ]; then
+	log "/usr/lib/choose-mirror/port_architecture is present"
+	is_ports_architecture=$(cat /usr/lib/choose-mirror/port_architecture)
+fi
+
#964248#142
Date:
2020-08-19 16:55:48 UTC
From:
To:
Le lundi 17 août à 00h 38mn 15s (+0200), John Paul Adrian Glaubitz a écrit :
Hi Adrian,

I think this is a misunderstanding.

There is no 'unreleased' suite on install cdrom
(and I am still unconvinced that one would be of any help there).

The option : --extra-suite=unreleased may be useful
when running debootstrap and installing from a
debian-ports mirror (ie through http or https, not
from a cdrom). This option shouldn't be used on
netinst and larger cdrom, only in netboot (and mini iso)
and businesscard.

Hence the test in the proposal [1] :

if [ "$PROTOCOL" = http ] || [ "$PROTOCOL" = https ] ||[ "$PROTOCOL" = ftp ]; then

I still hope it will help to find a solution.

Regards,
JH Chatenet

[1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=964248#134
#964248#147
Date:
2020-08-19 17:01:30 UTC
From:
To:
Le lundi 17 août à 00h 55mn 08s (+0200), John Paul Adrian Glaubitz a écrit :

Hi again,

I doubt that an empty 'unreleased' distribution is the cause
 of the problem (but an inexistent one causes an error with
--extra-suites=unreleased for sure).

hppa is an example of architecture with no regular deb packages in
'unreleased', only udeb. Trying :

# debootstrap --arch=hppa --variant=minbase --foreign --extra-suites=unreleased sid my_chroot http://ftp.de.debian.org/debian-ports
[ ... ]
: Extracting zlib1g...
0 dtl64-seine:/home/moi/cages# echo $?
0

So, it works (actually, I haven't tested in the context of the
debian-installer. I expect a similar result).

In the paricular case of hppa, --extra-suite is definitely useless but
seems harmless. Should it be enabled ?

Regards,
JH Chatenet
#964248#152
Date:
2020-08-19 17:04:30 UTC
From:
To:
Le lundi 17 août à 01h 06mn 05s (+0200), Samuel Thibault a écrit :

I am just curious : what are the expected benefits
of an 'unreleased' distribution on install cdrom ?

Regards,
JH Chatenet
#964248#157
Date:
2020-08-19 17:08:31 UTC
From:
To:
Le lundi 17 août à 10h 17mn 24s (+0200), John Paul Adrian Glaubitz a écrit :

Hi again,

At the time of writing (2020-08-19 16:00 UTC), it seems
that hurd-i386, powerpc and ppc64 are uninstallable from
'unstable' but would be with option : --extra-suites=unreleased

* hurd-i386 :

base package bsdmainutils(12.1.7) has a missing dependency :
 bsdextrautils (>= 2.35.2-7)
and bsdextrautils(2.35.2-7+hurd.1) is in 'unreleased'

required package libgnutls30(3.6.14-2+b1) has a missing dependency :
 libp11-kit0 (>= 0.23.18.1)
and libp11-kit0(0.23.18.1-2+hurd.2)

(actually, I am unsure of the installability of rsyslog)

* powerpc and ppc64 :

base package powerpc-utils(1.3.7-1) has a missing dependency :
 pmac-utils
and pmac-utils(1.1.3-27+ports) is in unreleased

I hope it will help !

Regards,
JH Chatenet
#964248#162
Date:
2020-08-19 17:04:51 UTC
From:
To:
There are individual packages from "unrelased" on the CD image when
building a CD with debian-cd because I added such a feature to debian-cd.

It was added like that to base-installer. Hence base-installer was run with
that argument when run during a CD installation.

And I already mentioned, "unreleased" is not really needed for most ports,
so I think it should rather be optional than default.

Adrian
#964248#167
Date:
2020-08-19 17:06:39 UTC
From:
To:
jhcha54008, le mer. 19 août 2020 19:04:30 +0200, a ecrit:

Err, containing the content of the unreleased distribution?

So that arch-specific packages are available, as well as patched
packages uploaded there (yes, I know, patches should be uploaded to
sid, but there's the maintainer latency there which hinders port
progression when not using unreleased as a staging area to be able to
move forward without having to wait).

Samuel
#964248#172
Date:
2020-08-19 17:27:30 UTC
From:
To:
Both powerpc and ppc64 are perfectly installable as pmac-utils is part
of the installation CD. I may also modify the powerpc-utils package
in the future to degrade pmac-utils to Recommends.

As I said, debian-cd allows packages from "unreleased" to be included
in the installation CD image.

Adrian
#964248#177
Date:
2020-08-19 17:46:23 UTC
From:
To:
If it works without issues (I mean if it doesn't abort during installation), then
I think the unreleased suite should be included for the hppa
architecture and all others by default. Specifically for hppa it could be
that we need to add packages in the future - even if it's best
to avoid adding packages there.

Helge
#964248#182
Date:
2020-08-21 11:11:26 UTC
From:
To:
Hi,

Thank you for your answer.


Le mercredi 19 août à 19h 06mn 39s (+0200), Samuel Thibault a écrit :


I agree that this is what the 'unreleased' distribution
on debian-ports mirrors is for.

But my question was : what about the install cdrom (and not
the debian-ports mirrors) ? What are the expected benefits of
keeping a separated 'unreleased' distribution on the cdrom ?

Currently, debian-cd (when setting UNRELEASED=1) is supposed
to pull all needed packages from both suites 'unstable' and
'unreleased', and to put everything in 'sid' on the built cdrom
image. Are there use cases where this scheme is unsuitable ?

Regards,
JH Chatenet
#964248#187
Date:
2021-02-20 00:34:07 UTC
From:
To:
Hello,

For information, I submitted a couple of fixes for the unreleased
distribution in debian-cd, it now behaves fine when setting UNRELEASED=1
in CONF.sh: it includes packages from the unreleased distribution in the
sid distribution of the CD. Also, I switched to using LOCAL=1,
LOCALDEBS and UPDATE_LOCAL=1, which is indeed enough to select any few
packages from unreleased for the bootstrap part.

Samuel