In my autopkgtest-virt-qemu test environment (in a sid VM),
debian/tests/lxc-old-testbed can no longer create an Ubuntu 14.04 'trusty'
container and is therefore skipped:

I believe this is because lxc-templates uses
/usr/share/keyrings/ubuntu-archive-removed-keys.gpg to get the
archive-signing key that was current at the time that Ubuntu 14.04 was
fully-supported, but ubuntu-keyring no longer ships that file since
2023.11.28.1-0.1.

As context for the cc'd maintainers, the purpose of this test is to make
sure that autopkgtest's ability to operate on extremely old testbeds
has not regressed. For the benefit of the Debian (E)LTS and Ubuntu ESM
teams, autopkgtest aims to be able to test Debian 8 'jessie' or newer and
Ubuntu 14.04 'trusty' or newer, but if we can no longer easily verify
this programmatically at release time, then that functionality should
be considered "at risk".

One way to solve this would be for ubuntu-keyring to provide
/usr/share/keyrings/ubuntu-archive-removed-keys.gpg again, analogous to
the way debian-archive-keyring provides a keyring of deprecated keys
(it doesn't need to be trusted, it just needs to exist); or for some
other appropriate package to provide a key that was used to sign trusty,
and for lxc-templates to use that key automatically when bootstrapping
a trusty container.

Another way to solve this would be to switch the lxc-old-testbed test to
construct a Debian 8 'jessie' testbed (presumably from archive.debian.org,
and possibly setting it to use an older archive-signing key).

If it would be easier to do this testing under an autopkgtest backend
that is not lxc (ideally podman or unshare) then that would also be fine.

Or, if the Debian (E)LTS and Ubuntu ESM teams are no longer interested in
jessie and trusty, or are willing to switch to using older versions of
autopkgtest to test those older suites, then we could bump up the minimum
system-under-test that we aim to support, which would also allow removing
various pieces of technical debt and legacy code from autopkgtest.

    smcv

Hello Simon,

Thanks! :)
jessie expired. APT in jessie is pretty stubborn about it, so our only
solution is to switch to the jessie-elts archive, signed with the
Freexian key. More on this issue the bottom of:
https://www.freexian.com/lts/extended/docs/how-to-use-extended-lts/

Maybe there's a way to use libfaketime to work-around this but I didn't
try ;)
2025-06-30 so it would be nice to keep this working. I believe
ci.freexian.com runs the same autopkgtest on all dists, e.g.:
https://ci.freexian.com/packages/r/rails/jessie/i386/91105/

Btw there was a similar discussion for debusine at:
https://salsa.debian.org/freexian-team/debusine/-/issues/459

Cheers!
Sylvain Beucler
Debian LTS Team

El 10/08/24 a las 15:53, Simon McVittie escribió:

Thank you! This is much appreciated.

Just FTR, the LTS/ELTS team is still interested in jessie (it will be
supported until 2025-06-30).

I obviously cannot talk for the Ubuntu security/ESM team, so I've added
them to CC, in case they want to answer. I just know that trusty is
planed to be supported until 2026-04:
https://canonical.com/blog/canonical-expands-long-term-support-to-12-years-starting-with-ubuntu-14-04-lts

Cheers,

Please could someone who works on Debian (E)LTS or Ubuntu ESM show their
appreciation by contributing a MR to make this test work again? :-)

As I said, if we don't have test coverage for extremely old testbeds, then
aiming to be able to test them is just an aim, and we can't be sure whether
it actually works.

I am not intending to work on this myself (I don't use suites this old and
am not paid to maintain them).

Thanks,
    smcv

Hi Simon, thanks for checking and reporting this.

We have this job running in autopkgtest CI, and it passes fine
while still printing the:

W: Cannot check Release signature; keyring file not available
/usr/share/keyrings/ubuntu-archive-removed-keys.gpg

warning. I tried reproducing the issue you describe locally on
a clean sid VM, and I am able to build the LXC trusty image
after installing the test deps for the "lxc" d/t/control test.
Again the "Cannot check Release signature" warning doesn't
appear to be a blocker. Terminal log excerpt:

------------------------------------------------------------------
root@debian-sid:~# dpkg-query -W autopkgtest
autopkgtest	5.39

root@debian-sid:~# autopkgtest-build-lxc ubuntu trusty amd64
lxc-create: autopkgtest-trusty-amd64: ../src/lxc/storage/btrfs.c: btrfs_create: 938 Inappropriate ioctl for device - Failed to create btrfs subvolume "/var/lib/lxc/autopkgtest-trusty-amd64/rootfs"
lxc-create: autopkgtest-trusty-amd64: ../src/lxc/storage/zfs.c: zfs_create: 742 Failed to create zfs dataset "zfs:lxc/autopkgtest-trusty-amd64": lxc-create: autopkgtest-trusty-amd64: ../src/lxc/utils.c: run_command_internal: 1813 Failed to exec command
lxc-create: autopkgtest-trusty-amd64: ../src/lxc/storage/lvm.c: do_lvm_create: 165 Failed to create logical volume "autopkgtest-trusty-amd64": lxc-create: autopkgtest-trusty-amd64: ../src/lxc/utils.c: run_command_internal: 1813 Failed to exec command
lxc-create: autopkgtest-trusty-amd64: ../src/lxc/storage/lvm.c: lvm_create: 623 Error creating new logical volume "lvm:/dev/lxc/autopkgtest-trusty-amd64" of size "1073741824 bytes"
Checking cache download in /var/cache/lxc/trusty/rootfs-amd64 ...
Installing packages in template: eatmydata,language-pack-en
Downloading ubuntu trusty minimal ...
W: Cannot check Release signature; keyring file not available /usr/share/keyrings/ubuntu-archive-removed-keys.gpg
I: Retrieving InRelease
I: Retrieving Release
I: Retrieving Packages
I: Validating Packages
I: Retrieving Packages
I: Validating Packages
I: Resolving dependencies of required packages...
I: Resolving dependencies of base packages...
I: Checking component main on http://archive.ubuntu.com/ubuntu...
I: Retrieving adduser 3.113+nmu3ubuntu3

[...]

Unpacking linux-headers-generic (3.13.0.170.181) ...
Setting up linux-headers-3.13.0-170 (3.13.0-170.220) ...
Setting up linux-headers-3.13.0-170-generic (3.13.0-170.220) ...
Setting up linux-headers-generic (3.13.0.170.181) ...
root@debian-sid:~# echo $?
0
------------------------------------------------------------------

The autopkgtest-build-lxc tool tries to do some proxy autoconfiguration,
the logic is in lib/build-lxc.sh. I remember you mentioning using
apt-cacher-ng, and having issues reaching archive.ubuntu.com with that,
despite /var/lib/apt-cacher-ng/backends_ubuntu.default allowing access
to archive.ubuntu.com by default. My guess is that the culprit is in
this area.

I found:

https://bugs.launchpad.net/ubuntu/+source/apt-cacher-ng/+bug/1998865

which describes a similar problem and mentions possible workarounds.

Hi,

FTR I referenced this issue in the ELTS internal tracker.

(I didn't see follow-ups to this bug until I checked again by chance;
honestly I'm still puzzled on how the BTS is supposed to help with that.)

Cheers!
Sylvain Beucler
Debian LTS Team

On Mon, 19 Aug 2024 14:46:05 +0200 Paride Legovini <paride@debian.org> wrote:
[...]


Also: lxc-old-testbed appears to pass fine in debci, e.g.

https://ci.debian.net/packages/a/autopkgtest/testing/amd64/50481379/

110s autopkgtest [04:15:56]: testing package autopkgtest version 5.39
[...]
8383s lxc-old-testbed      PASS
[...]
12554s autopkgtest [07:43:20]: @@@@@@@@@@@@@@@@@@@@ summary
12554s autopkgtest          PASS
12554s installed            PASS
12554s docker               PASS
12554s lxc                  PASS
12554s lxc-old-testbed      PASS
12554s lxd                  PASS
12554s podman               PASS
12554s podman-init          PASS
12554s schroot              PASS
12554s unshare              PASS

Again lxc-old-testbed does not seem to be broken to me.

Bootstrapping old Debian and Ubuntu releases can also fail if you are
using gpgv-from-sq, see #1076358.

Hi,
For the record, jessie-elts is EOL since 2025-07-01 so there's no more
effort on that release.

I proposed a debci patch to keep supporting buster though (which broke
after its archival):
https://salsa.debian.org/ci-team/debci/-/merge_requests/298

Cheers!
Sylvain Beucler
Debian LTS Team

Es gibt eine Familienspende in Höhe von 1.850.000,00 USD von Cheng Charlie
Saephan. Bitte antworten Sie für weitere Informationen. Denken Sie daran,
Ihrer Familie und den Bedürftigen in Ihrer Umgebung Gutes zu tun.

Dies ist bereits der zweite Versuch, Sie zu erreichen. Bitte antworten Sie
für weitere Details.