The CI jobs scheduled by britney don't take transitions into account, causing not all affected packages to be pulled from unstable.

As recent discussed in the gdal transition bugreport [0], CI job scheduling needs to be improved to do the right thing for packages with multiple packages affected by ongoing transitions like libgdal-grass.

The attached script is a proof of concept that implements the algorithm I suggested to solve this issue.

It parses the list of dependencies for the autopkgtests and expands this list to include all transitive dependencies.

The list of affected source packages in the transition for the trigger package is parsed from the Ben JSON output files, the state of the transition is determined by the location of the .ben file (ongoing & finished subdirectories).

For the transitive test dependencies, if their source package is affected by the transition, that source package is pinned in the CI job.

The debci API is not used currently, output is printed which can be copy/pasted into the self-service form which should suffice for a POC.

The dependency resolution currently uses UDD, because I didn't yet figure out how to use python-apt's apt_pkg to do a Trivial-Only run in forky chroot on a trixie system, pointers are very welcome.

Example output for two different packages from the recent gdal transition:

 CI Job Request

 Package Name: libgeo-gdal-ffi-perl
 Suite: testing
 Trigger: gdal/3.12.0+dfsg-1

 Pin Packages:
 package_version: gdal/3.12.0+dfsg-1
 package: gdal
 src:gdal, unstable
 src:libgeo-gdal-ffi-perl, unstable

 Extra APT Sources:
 unstable
 testing

This only pulls gdal & libgeo-gdal-ffi-perl from unstable because non of the transitive dependency were also involved in the transtion.

 CI Job Request

 Package Name: libgdal-grass
 Suite: testing
 Trigger: gdal/3.12.0+dfsg-1

 Pin Packages:
 package_version: gdal/3.12.0+dfsg-1
 package: gdal
 src:gdal, unstable
 src:grass, unstable
 src:libgdal-grass, unstable

 Extra APT Sources:
 unstable
 testing

Here all three affected packages are pulled from unstable which is the only combination that will work as discussed in all the recent gdal transition bugreports.

I guess when dependency resolution is implemented with python-apt, this should get incorporated in britney2?

How difficult is it to setup a test instance for that? setting-up-britney.rst doesn't look very daunting, but I suspect it leave out a lot of details.

Before I spend more time on this, I'd like to hear what you think of this approach.

[0] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1120361#53

Kind Regards,

Bas

Hi Bas,

Thanks for your work on this.

britney2 uses apt_pkg indeed, loading the archive is done in
britney2/utils.py and britney2/inputs/suiteloader.py


I have a test instance of britney2 on respighi in ~elbrus/britney2. I
created some directories with the right symlinks to the live data and
the rest I handle in ~elbrus/bin/britney-elbrus, which is mostly about
copying state from the real britney instance to some directories where
my instance can write updates and to drive britney2 with my
configuration (which is mainly a delta in directories and occasionally
new configuration I want to try).

Yes and no. It leaves out details on how you can further change the
behavior of britney with its configuration, but it seems most of the
setup is described.

One thing that's important to realize is that britney2 is meant to not
do any collecting of data by itself. All data sources have to be
available on respighi, either via a mount by DSA or pre-fetched by
britney1 (which is supposed to be rather dumb). As I think britney2
already has nearly all the information internalized that is needed, it
could do a guess at which transitions are ongoing, very similar to the
auto-transition script and calculate the involved packages from there.

I think I spotted that you are parsing the d/t/control file. Currently
we don't have those available on respighi (and I don't think we need
this for your solution to work).

Paul

You're welcome.

Thanks, I'll have a look at those and have another stab at getting apt to do the dependency resolution.

I'll have a look at that as well.
I don't see how we can do without d/t/control to get the test dependencies, those tend to be similar to Build-Depends and Depends but they are not the same, and the test dependencies are not available in Sources.gz or Packages.gz

I guess you're suggesting to just limit the dependency resolution to the binary packages of the source package to be tested using Depends and Recommends from Packages.gz, that likely covers most but not autopkgtests.

Kind Regards,

Bas

Hi


They are added to Sources via the Testsuite-Triggers field, which exists
for this purpose. And yes, some information is stripped that way. There
are multiple reasons why I'd want britney2 to have access to all
d/t/control stanza (or at least a richer data set than
Testsuite-Triggers) but I don't think it's needed for this bug.

Paul

Hi


They are added to Sources via the Testsuite-Triggers field, which exists
for this purpose. And yes, some information is stripped that way. There
are multiple reasons why I'd want britney2 to have access to all
d/t/control stanza (or at least a richer data set than
Testsuite-Triggers) but I don't think it's needed for this bug.

Paul

Using python-apt's apt_pkg with a chroot turns out to be non-trivial unlike apt.cache.

I'm running a script [0] on a trixie system and need to interact with the cache in a testing chroot.

The chroot is created with mmdebstrap --mode=unshare, and the script is run via unshare as well.

Setting RootDir in the config does not result in the chroot paths to be used:

  $ unshare --map-auto --map-user=65536 --map-group=65536 --keep-caps -- ./bin/resolve-dependencies.py -vdp opencv -u 2>&1 | tee /tmp/resolve-dependencies.log
  Resolving dependencies for: opencv
  Chroot already exists: /var/tmp/testing-chroot
  config:
  RootDir "/var/tmp/testing-chroot";

  list:
  [<apt_pkg.MetaIndex object: type='deb', uri:'http://ftp.nl.debian.org/debian/' dist='trixie' is_trusted='1'>,
   <apt_pkg.MetaIndex object: type='deb', uri:'http://ftp.nl.debian.org/debian/' dist='trixie-updates' is_trusted='1'>,
   <apt_pkg.MetaIndex object: type='deb', uri:'http://security.debian.org/debian-security/' dist='trixie-security' is_trusted='1'>,
   <apt_pkg.MetaIndex object: type='deb', uri:'http://ftp.nl.debian.org/debian/' dist='trixie-backports' is_trusted='1'>,
   <apt_pkg.MetaIndex object: type='deb', uri:'http://debug.mirrors.debian.org/debian-debug/' dist='trixie-debug' is_trusted='1'>]
  Reading package lists... Done
  Building dependency tree... Done
  Reading state information... Done
  Traceback (most recent call last):
    File "/home/bas/git/release-team/release.debian.org/bin/resolve-dependencies.py", line 216, in <module>
      sys.exit(main())
               ~~~~^^
    File "/home/bas/git/release-team/release.debian.org/bin/resolve-dependencies.py", line 212, in main
      return resolve_dependencies()
    File "/home/bas/git/release-team/release.debian.org/bin/resolve-dependencies.py", line 97, in resolve_dependencies
      cache.update(apt.progress.text, source_list)
      ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  apt_pkg.Error: E:Could not open lock file /var/lib/apt/lists/lock - open (13: Permission denied), E:Unable to lock directory /var/lib/apt/lists/

How can apt_pkg be used with a chroot?

apt.Cache(rootdir) works, but it does not provide access to source packages like apt_pkg.SourceRecords.

[0] https://salsa.debian.org/sebastic/release.debian.org/-/blob/ce71f7e543283c7fcca0a701fe22667ae7a188a2/bin/resolve-dependencies.py

Kind Regards,

Bas

Please CC me as I'm not subscribed to the list.

Hi,

Quoting Sebastiaan Couwenberg (2025-12-02 18:36:00)

maybe this is an XY problem.

Why do you need a full chroot for something that is as far as I get from the
script just a "I want apt to resolve the dependencies with my custom
sources.list"? This sounds like something like chdist is doing?

Maybe the mmdebstrap man page has an example which helps you. It is a snippet
which is the very bare-bones of what mmdebstrap does with a few thousand more
lines of code:


    mkdir -p "$2/etc/apt" "$2/var/cache" "$2/var/lib"
    cat << END > "$2/apt.conf"
    Apt::Architecture "$(dpkg --print-architecture)";
    Apt::Architectures "$(dpkg --print-architecture)";
    Dir "$(cd "$2" && pwd)";
    Dir::Etc::Trusted "$(eval "$(apt-config shell v Dir::Etc::Trusted/f)"; printf %s "$v")";
    Dir::Etc::TrustedParts "$(eval "$(apt-config shell v Dir::Etc::TrustedParts/d)"; printf %s "$v"

    END
    echo "deb http://deb.debian.org/debian/ $1 main" > "$2/etc/apt/sources.list"
    APT_CONFIG="$2/apt.conf" apt-get update
    APT_CONFIG="$2/apt.conf" apt-get --yes --download-only install '?essential'
    for f in "$2"/var/cache/apt/archives/*.deb; do dpkg-deb --extract "$f" "$2"; done
    chroot "$2" sh -c "dpkg --install --force-depends /var/cache/apt/archives/*.deb"

I think you need Dir.

This sounds like you are trying to access something created for the unshared
user as the normal user?

Thanks!

cheers, josch

I want to use apt to resolve dependencies for packages (both source & binary) in testing (and unstable), but the code won't be running on a testing (or unstable) system.

See #1120799 for context.

How would python-apt's apt_pkg use this?

Not according to the apt.conf(5):

"
   The configuration item RootDir has a special meaning. If set, all paths will be relative to RootDir, even paths that are specified absolutely. So, for instance, if RootDir is set to /tmp/staging and Dir::State::status is set to /var/lib/dpkg/status, then the status file will be looked up in /tmp/staging/var/lib/dpkg/status. If you want to prefix only relative paths, set Dir instead.
"

Using Dir fails in the same way:

  $ unshare --map-auto --map-user=65536 --map-group=65536 --keep-caps -- ~/git/release-team/release.debian.org/bin/resolve-dependencies.py -vdp opencv -u 2>&1 | tee /tmp/resolve-dependencies.log
  Resolving dependencies for: opencv
  Chroot already exists: /var/tmp/testing-chroot
  config:
  Dir "/var/tmp/testing-chroot";

  list:
  [<apt_pkg.MetaIndex object: type='deb', uri:'http://ftp.nl.debian.org/debian/' dist='trixie' is_trusted='1'>,
   <apt_pkg.MetaIndex object: type='deb', uri:'http://ftp.nl.debian.org/debian/' dist='trixie-updates' is_trusted='1'>,
   <apt_pkg.MetaIndex object: type='deb', uri:'http://security.debian.org/debian-security/' dist='trixie-security' is_trusted='1'>,
   <apt_pkg.MetaIndex object: type='deb', uri:'http://ftp.nl.debian.org/debian/' dist='trixie-backports' is_trusted='1'>,
   <apt_pkg.MetaIndex object: type='deb', uri:'http://debug.mirrors.debian.org/debian-debug/' dist='trixie-debug' is_trusted='1'>]
  Reading package lists... Done
  Building dependency tree... Done
  Reading state information... Done
  Traceback (most recent call last):
    File "/home/bas/git/release-team/release.debian.org/bin/resolve-dependencies.py", line 216, in <module>
      sys.exit(main())
               ~~~~^^
    File "/home/bas/git/release-team/release.debian.org/bin/resolve-dependencies.py", line 212, in main
      return resolve_dependencies()
    File "/home/bas/git/release-team/release.debian.org/bin/resolve-dependencies.py", line 97, in resolve_dependencies
      cache.update(apt.progress.text, source_list)
      ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  apt_pkg.Error: E:Could not open lock file /var/lib/apt/lists/lock - open (13: Permission denied), E:Unable to lock directory /var/lib/apt/lists/

No, this is python-apt running outside the chroot as also hinted by the sources.list for trixie.

Running the script directory instead of via unshare fails in the same way because my unprivileged user is not allowed to write those directories:

  $ ~/git/release-team/release.debian.org/bin/resolve-dependencies.py -vdp opencv -u 2>&1 | tee /tmp/resolve-dependencies.log
  Resolving dependencies for: opencv
  Chroot already exists: /var/tmp/testing-chroot
  config:
  RootDir "/var/tmp/testing-chroot";

  list:
  [<apt_pkg.MetaIndex object: type='deb', uri:'http://ftp.nl.debian.org/debian/' dist='trixie' is_trusted='1'>,
   <apt_pkg.MetaIndex object: type='deb', uri:'http://ftp.nl.debian.org/debian/' dist='trixie-updates' is_trusted='1'>,
   <apt_pkg.MetaIndex object: type='deb', uri:'http://security.debian.org/debian-security/' dist='trixie-security' is_trusted='1'>,
   <apt_pkg.MetaIndex object: type='deb', uri:'http://ftp.nl.debian.org/debian/' dist='trixie-backports' is_trusted='1'>,
   <apt_pkg.MetaIndex object: type='deb', uri:'http://debug.mirrors.debian.org/debian-debug/' dist='trixie-debug' is_trusted='1'>]
  Reading package lists... Done
  Building dependency tree... Done
  Reading state information... Done
  Traceback (most recent call last):
    File "/home/bas/git/release-team/release.debian.org/bin/resolve-dependencies.py", line 216, in <module>
      sys.exit(main())
               ~~~~^^
    File "/home/bas/git/release-team/release.debian.org/bin/resolve-dependencies.py", line 212, in main
      return resolve_dependencies()
    File "/home/bas/git/release-team/release.debian.org/bin/resolve-dependencies.py", line 97, in resolve_dependencies
      cache.update(apt.progress.text, source_list)
      ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  apt_pkg.Error: E:Could not open lock file /var/lib/apt/lists/lock - open (13: Permission denied), E:Unable to lock directory /var/lib/apt/lists/

Running the script as root does not fail on update, but it works with the trixie data instead of the testing:

  $ su -c "~bas/git/release-team/release.debian.org/bin/resolve-dependencies.py -vdp opencv -u" 2>&1 | tee /tmp/resolve-dependencies.log
  Password:
  Resolving dependencies for: opencv
  Chroot already exists: /var/tmp/testing-chroot
  config:
  RootDir "/var/tmp/testing-chroot";

  list:
  [<apt_pkg.MetaIndex object: type='deb', uri:'http://ftp.nl.debian.org/debian/' dist='trixie' is_trusted='1'>,
   <apt_pkg.MetaIndex object: type='deb', uri:'http://ftp.nl.debian.org/debian/' dist='trixie-updates' is_trusted='1'>,
   <apt_pkg.MetaIndex object: type='deb', uri:'http://security.debian.org/debian-security/' dist='trixie-security' is_trusted='1'>,
   <apt_pkg.MetaIndex object: type='deb', uri:'http://ftp.nl.debian.org/debian/' dist='trixie-backports' is_trusted='1'>,
   <apt_pkg.MetaIndex object: type='deb', uri:'http://debug.mirrors.debian.org/debian-debug/' dist='trixie-debug' is_trusted='1'>]
  Reading package lists... Done
  Building dependency tree... Done
  Reading state information... Done
  Error in function start
  TypeError: AcquireProgress.start() missing 1 required positional argument: 'self'
  [...]
  Error in function stop
  TypeError: AcquireProgress.stop() missing 1 required positional argument: 'self'
  Package: opencv
  Version: 4.10.0+dfsg-5
  Build-Depends: [...]

This the version in trixie, I need the data for forky:

  $ rmadison -s stable,testing opencv
  opencv     | 4.10.0+dfsg-5 | stable     | source
  opencv     | 4.10.0+dfsg-6 | testing    | source

Kind Regards,

Bas

This seems to be an issue with python-apts apt_pkg, because Perl AptPkg works as expected:

  $ unshare --map-auto --map-user=65536 --map-group=65536 --keep-caps -- ~/git/release-team/release.debian.org/bin/resolve-dependencies.pl
  Package: opencv
  Version: 4.10.0+dfsg-6
  Source:  https://deb.debian.org/debian/pool/main/o/opencv/opencv_4.10.0%2bdfsg-6.dsc

This Perl script [1] is getting the data for testing by using the chroot.

[1] https://salsa.debian.org/sebastic/release.debian.org/-/blob/a6708dcc8e00ce1e16b1cd94909890e8098ea2f5/bin/resolve-dependencies.pl

Kind Regards,

Bas

python-apt fails to use the chdist the same way it fails to use the chroot, Perl AptPkg uses both as expected.

You can reproduce this with the scripts from my transition-autopkgtest branch:

https://salsa.debian.org/sebastic/release.debian.org/-/commit/1d872c5a1d30ce5bea2064de75747ea4b6f0e73b

Perl AptPkg initializes the system via the config:

  $_config->init;
  if ($cfg{use_chdist}) {
      $_config->set('Dir', $cfg{chdist_path});
  }
  else {
      $_config->set('RootDir', $cfg{chroot_path});
  }
  $_system = $_config->system;

Whereas python-apt initializes them separately:

  apt_pkg.init_config()
  apt_pkg.init_system()

  config = apt_pkg.Configuration()
  if args.use_chdist:
      config.set('Dir', args.chdist_path)
  else:
      config.set('RootDir', args.chroot_path)

The modified config is not used by the python-apt classes.

The config dump for Perl AptPkg shows a mix of the system apt.conf and the config set in the code, the config dump for python-apt shows only the config set in the code.

Kind Regards,

Bas

That's because apt_pkg.config needs to be used instead of another apt_pkg.Configuration instance:

https://salsa.debian.org/sebastic/release.debian.org/-/commit/d7428cd396313d20773d0f10331623f1e2cb2ff6

Now both the Perl & Python scripts work as expected with the chroot & chdist:

  # Python with chroot
  $ unshare --map-auto --map-user=65536 --map-group=65536 --keep-caps -- ~/git/release-team/release.debian.org/bin/resolve-dependencies.py -vdp opencv
  Resolving dependencies for: opencv
  Chroot already exists: /var/tmp/testing-chroot
  config:
  [...]
  Dir "/";
  [...]
  RootDir "/var/tmp/testing-chroot";

  list:
  [<apt_pkg.MetaIndex object: type='deb', uri:'https://deb.debian.org/debian/' dist='testing' is_trusted='1'>]
  Reading package lists... Done
  Building dependency tree... Done
  Reading state information... Done
  Package: opencv
  Version: 4.10.0+dfsg-6
  [...]


  # Python with chdist
  $ ~/git/release-team/release.debian.org/bin/resolve-dependencies.py -vdp opencv -D
  Resolving dependencies for: opencv
  config:
  [...]
  Dir "/var/tmp/testing-chdist";
  [...]

  list:
  [<apt_pkg.MetaIndex object: type='deb', uri:'https://deb.debian.org/debian/' dist='testing' is_trusted='1'>]
  Reading package lists... Done
  Building dependency tree... Done
  Package: opencv
  Version: 4.10.0+dfsg-6
  [...]


  # Perl with chroot
  $ unshare --map-auto --map-user=65536 --map-group=65536 --keep-caps -- ~/git/release-team/release.debian.org/bin/resolve-dependencies.pl -vdp opencv
  config:
  [...]
  Dir "/";
  [...]
  RootDir "/var/tmp/testing-chroot";
  Getting source package: opencv
  Package: opencv
  Version: 4.10.0+dfsg-6
  Source:  https://deb.debian.org/debian/pool/main/o/opencv/opencv_4.10.0%2bdfsg-6.dsc


  # Perl with chdist
  $ ~/git/release-team/release.debian.org/bin/resolve-dependencies.pl -vdp opencv -D
  config:
  [..]
  Dir "/var/tmp/testing-chdist";
  [...]
  Getting source package: opencv
  Package: opencv
  Version: 4.10.0+dfsg-6
  Source:  https://deb.debian.org/debian/pool/main/o/opencv/opencv_4.10.0%2bdfsg-6.dsc


Kind Regards,

Bas

The best we can do without parsing autodep8 output is to mimic ben and resolve the dependencies for the arch:any binary packages of the source package in question.

Kind Regards,

Bas