Seems the latest version of python-apt has some serious regressions as
https://tracker.debian.org/pkg/python-apt shows wide-spread
autopkgtest failures.

In a clean Debian unstable container a simple installation is failing with:

Hit:1 http://deb.debian.org/debian sid InRelease
Get:2 http://deb.debian.org/debian sid/main all Contents (deb) [43.5 MB]
Get:3 http://deb.debian.org/debian sid/main amd64 Contents (deb) [15.8 MB]
Traceback (most recent call last):
  File "/usr/lib/cnf-update-db", line 3, in <module>
    import apt_pkg
ImportError: /usr/lib/python3/dist-packages/apt_pkg.cpython-312-x86_64-linux-gnu.so:
undefined symbol:
_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm,
version APTPKG_6.0
Fetched 59.3 MB in 13s (4620 kB/s)
Reading package lists...
E: Problem executing scripts APT::Update::Post-Invoke-Success 'if
/usr/bin/test -w /var/lib/command-not-found/ -a -e
/usr/lib/cnf-update-db; then /usr/lib/cnf-update-db > /dev/null; fi'
E: Sub-process returned an error code
Error: error building at STEP "RUN apt-get install -q --yes
command-not-found &&     apt-get update -q": error while running
runtime: exit status 100

Please take a look what is going on.






Please also consider using better CI before uploads.

I don't know if the root cause of these are the same, but a simple
Salsa CI test run at
https://salsa.debian.org/otto/python-apt/-/pipelines/769109 errors on:

dpkg-buildpackage: info: source package python-apt
dpkg-buildpackage: info: source version 2.9.1+salsaci+20241126+1
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by Julian Andres Klode <jak@debian.org>
 dpkg-source --before-build .
dpkg-buildpackage: info: host architecture amd64
 debian/rules clean
dh clean --with python3,sphinxdoc --buildsystem=pybuild
   dh_auto_clean -O--buildsystem=pybuild
I: pybuild base:311: python3.13 setup.py clean
Traceback (most recent call last):
  File "/builds/otto/python-apt/debian/output/source_dir/setup.py",
line 141, in <module>
    setup(
    ~~~~~^
        name="python-apt",
        ^^^^^^^^^^^^^^^^^^
    ...<15 lines>...
        platforms="posix",
        ^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line
117, in setup
    return distutils.core.setup(**attrs)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/usr/lib/python3/dist-packages/setuptools/_distutils/core.py",
line 145, in setup
    _setup_distribution = dist = klass(attrs)
                                 ~~~~~^^^^^^^
  File "/usr/lib/python3/dist-packages/setuptools/dist.py", line 333,
in __init__
    self.metadata.version = self._normalize_version(self.metadata.version)
                            ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/setuptools/dist.py", line 369,
in _normalize_version
    normalized = str(Version(version))
                     ~~~~~~~^^^^^^^^^
  File "/usr/lib/python3/dist-packages/packaging/version.py", line
202, in __init__
    raise InvalidVersion(f"Invalid version: {version!r}")
packaging.version.InvalidVersion: Invalid version: '2.9.1+salsaci+20241126+1'
E: pybuild pybuild:389: clean: plugin distutils failed with: exit
code=1: python3.13 setup.py clean
dh_auto_clean: error: pybuild --clean -i python{version} -p "3.13
3.12" returned exit code 13
make: *** [debian/rules:17: clean] Error 25
dpkg-buildpackage: error: debian/rules clean subprocess returned exit status 2


If you want, I can submit a MR for you to enable Salsa CI in
apt-python. However I wanted to check first if you want help, as in my
previous submissions you have been rather dismissive to getting
assistance and bug fixes
(https://salsa.debian.org/jak/command-not-found/-/merge_requests/8,
https://salsa.debian.org/apt-team/apt/-/merge_requests/348).

Perhaps some pre-depends relationship is missing and upgrades fail due
to mismatching ABI..?

I am only seeing this issue on upgrades. Bootstrapping a system from
scratch does not have this issue.

(Dropping the CCs, also use X-Debbugs-Cc otherwise the thread
 splits in two)

This is highly awkward, that is a C++ standard library symbol:

std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> >::_M_replace(unsigned long, unsigned long, char
const*, unsigned long)

It's in a template, and we build all inline functions with hidden
visibility in APT; so it's in a sense expected, but also unexpected,
as everything that uses the function keeps its own copy of it.

But I suppose this is an issue because other versions don't have
the versioning tag and the linker accidentally picked a symbol that
I don't know, doesn't really exist? I don't know how this happens.
installed test suite run in autopkgtest, and they of course all
import that module, so I don't know how we can get into that
spot, but I can reproduce it.

Clean container (your example is from a system with command-not-found),
install python3-apt, and then

root@fecbc40d24e5:/# python3 -c "import apt_pkg"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: /usr/lib/python3/dist-packages/apt_pkg.cpython-312-x86_64-linux-gnu.so: undefined symbol: _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm, version APTPKG_6.0

That's to be expected, yes, package versions need to conform not just
to Debian's version format but also the Python one, so the version
picked by salsa-ci is wrong.

So I think we need some editing there to tell salsa-ci to use a
version format that works.

I am making progress on the apt merge request, but I need to put my
thoughts into words. I was doing research on that yesterday. But you
can see there is _a lot_ happening and it's not the hugest priority
- We can improve the CI after a freeze but we can't land or remove
features after it. Specifically the points to consider right now
are "enable this without other gitlabs failing" and "enable it for
any commit that touches debian/ or CMakeLists.txt".

The command-not-found one was an absurd confusion on my side when
I was rushing to fix command-not-found and didn't find an email
and forgot to look at merge requests.

I'd be happy to accept patches to add a debian/salsa-ci.yml
for python-apt. I thought about doing it myself yesterday.

The tests were skipped at build time, d'oh!

I: pybuild base:311: env PYTHONPATH=/<<PKGBUILDDIR>>/.pybuild/cpython3_3.13_apt/build python3.13 tests/test_all.py
[tests] Skipping because sources.list is not readable

Some of them are just smoke tests relying on the host's APT
lists and I suppose they don't exist anymore.

I wonder if the same thing happened in my autopkgtest.

It seems to be an issue in symbols; the shlibs get set to

libapt-pkg6.0t64 >= 2.7.11

but the symbol is not present in the outdated container (2.9.4
is what my debian:unstable OCI comes with). Upgrading libapt-pkg6.0t64
to 2.9.14 fixes the issue.

But noticably, we can check our symbols file for apt:

    $ grep "_M_replace" debian/libapt-pkg6.0t64.symbols

It's empty, so the dependency should have been generated against
the latest version if anything.

Or rather the problem here is that we are linking this in the
first place, inline functions should be resolved locally, they
are hidden in APT.

I wonder if toolchain people/doko has a clue, because I don't.
The way we built APT is:

Control: clone -1 -2 -3
Control: reassign -2 apt: -Wp,-D_GLIBCXX_ASSERTION defines new symbols
Control: severity -2 important
Control: reassign -3 g++-14
Control: retitle -3 g++-14: Linking picks custom library symbols over stdc++
Control: severity -3 normal

std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_replace(unsigned long, unsigned long, char const*, unsigned long)@@APTPKG_6.0

aka std::string::replace().

In APT 2.9.4 that is being tested, here, the symbol is undefined and referencing the version shipped
in libstdc++:

root@84857ed287c5:/# grep _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm <(nm -D /usr/lib/x86_64-linux-gnu/libapt-pkg.so.6.0)
                 U _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm@GLIBCXX_3.4.21

However in 2.9.14, the symbol is suddenly defined and hence emitted using the APTPKG_6.0 versioning per our

	APTPKG_6.0 { global: *; };

version script:

# grep _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm <(nm -D /usr/lib/x86_64-linux-gnu/libapt-pkg.so.6.0)
000000000009b1f0 W _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm@@APTPKG_6.0

This now caused python-apt to favour that version of the symbol over the C++
one when it called std::string::replace() itself. Normally if we did not have
that version script, all callers of an inline function should receive a weak
symbol and one is resolved at runtime - however, here the linker decided to
instead drop the weak symbol in favor of using the export from APTPKG_6.0;
the inverse of what happened in libapt-pkg6.0t64 itself.

This is all not correct: The expectation for the symbols provided by GLIBCXX_
versions is to have them be satisfied by libstdc++, so you don't need to
keep around multiple copies of common things like std::string::replace().

Rebuilding apt 2.9.14 in testing does not restore the symbol to undefined,
so it must be down to a change in apt. And there is one: In the second object
that is linked in, a call to std::string::replace has been dropped. Is
it that?

So now we build 2.9.4 and 2.9.14 and get:

root@2540807186ac:/build/apt# nm -D obj-x86_64-linux-gnu/apt-pkg/libapt-pkg.so | grep _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm
                 U _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm@GLIBCXX_3.4.21
root@2540807186ac:/build/apt# cp ^C
root@2540807186ac:/build/apt# cp -a obj-x86_64-linux-gnu/ 2.9.4
root@2540807186ac:/build/apt# git checkout 2.9.14
Previous HEAD position was de7c78680 Release 2.9.4
HEAD is now at 1a03ebc11 Release 2.9.14
root@2540807186ac:/build/apt# ninja -C obj-x86_64-linux-gnu/ apt-pkg
ninja: Entering directory `obj-x86_64-linux-gnu/'
[0/1] Re-running CMake...
-- Performing Test have-compiler-flag:-Wp,-D_GLIBCXX_ASSERTIONS
-- Performing Test have-compiler-flag:-Wp,-D_GLIBCXX_ASSERTIONS - Success
-- Detected vendor: debian (cached)
-- Building libapt-pkg 6.0 (release 0)
-- Could NOT find GTest (missing: GTEST_LIBRARY GTEST_INCLUDE_DIR GTEST_MAIN_LIBRARY) 
-- Found GTest at /usr/src/googletest/googletest, headers at /usr/src/googletest/googletest/include
-- Configuring done (0.4s)
-- Generating done (0.1s)
-- Build files have been written to: /build/apt/obj-x86_64-linux-gnu
[67/68] Creating library symlink apt-pkg/libapt-pkg.so.6.0 apt-pkg/libapt-pkg.so
root@2540807186ac:/build/apt# nm -D obj-x86_64-linux-gnu/apt-pkg/libapt-pkg.so | grep _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm
0000000000089f60 W _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm@@APTPKG_6.0


Hmm but oh  Let's revert -Wp,-D_GLIBCXX_ASSERTIONS and we get:

root@2540807186ac:/build/apt# ninja -C obj-x86_64-linux-gnu/ apt-pkg
ninja: Entering directory `obj-x86_64-linux-gnu/'
[0/1] Re-running CMake...
-- Detected vendor: debian (cached)
-- Building libapt-pkg 6.0 (release 0)
-- Could NOT find GTest (missing: GTEST_LIBRARY GTEST_INCLUDE_DIR GTEST_MAIN_LIBRARY) 
-- Found GTest at /usr/src/googletest/googletest, headers at /usr/src/googletest/googletest/include
-- Configuring done (0.3s)
-- Generating done (0.1s)
-- Build files have been written to: /build/apt/obj-x86_64-linux-gnu
[67/68] Creating library symlink apt-pkg/libapt-pkg.so.6.0 apt-pkg/libapt-pkg.so
root@2540807186ac:/build/apt# nm -D obj-x86_64-linux-gnu/apt-pkg/libapt-pkg.so | grep _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm
                 U _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm@GLIBCXX_3.4.21


So what happens is that the linker is deduplicating the symbol,
preferring the @GLIBCXX_3.4.21 version when it's the same, but
now we built with assertions, so it's not the same anymore,
so it's not dedupped.

It's unclear why when python-apt is built the linker prefers libapt-pkg's
version over libstdc++

And of course we used to have these symbols pop up before we
used -fvisibility=hidden, so it turns out that they had an
entry in the symbols file using a regex to match, that was
set to 0.8.0, and hence python-apt got the wrong version.


But what is going on in python-apt itself now? It is confronted
with the symbol from libstdc++ and libapt-pkg.

Building the normal way we get the APTPKG_6.0 symbol:

x86_64-linux-gnu-g++ -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 build/temp.linux-x86_64-cpython-312/python/acquire-item.o build/temp.linux-x86_64-cpython-312/python/acquire.o build/temp.linux-x86_64-cpython-312/python/apt_pkgmodule.o build/temp.linux-x86_64-cpython-312/python/cache.o build/temp.linux-x86_64-cpython-312/python/cachegroup.o build/temp.linux-x86_64-cpython-312/python/cdrom.o build/temp.linux-x86_64-cpython-312/python/configuration.o build/temp.linux-x86_64-cpython-312/python/depcache.o build/temp.linux-x86_64-cpython-312/python/generic.o build/temp.linux-x86_64-cpython-312/python/hashes.o build/temp.linux-x86_64-cpython-312/python/hashstring.o build/temp.linux-x86_64-cpython-312/python/hashstringlist.o build/temp.linux-x86_64-cpython-312/python/indexfile.o build/temp.linux-x86_64-cpython-312/python/lock.o build/temp.linux-x86_64-cpython-312/python/metaindex.o build/temp.linux-x86_64-cpython-312/python/orderlist.o build/temp.linux-x86_64-cpython-312/python/pkgmanager.o build/temp.linux-x86_64-cpython-312/python/pkgmanagerprogress.o build/temp.linux-x86_64-cpython-312/python/pkgrecords.o build/temp.linux-x86_64-cpython-312/python/pkgsrcrecords.o build/temp.linux-x86_64-cpython-312/python/policy.o build/temp.linux-x86_64-cpython-312/python/progress.o build/temp.linux-x86_64-cpython-312/python/python-apt-helpers.o build/temp.linux-x86_64-cpython-312/python/sourcelist.o build/temp.linux-x86_64-cpython-312/python/string.o build/temp.linux-x86_64-cpython-312/python/tag.o -L/usr/lib/x86_64-linux-gnu -lapt-pkg -o build/lib.linux-x86_64-cpython-312/apt_pkg.cpython-312-x86_64-linux-gnu.so
root@79f30361847b:/build/python-apt# nm -D  build/lib.linux-x86_64-cpython-312/apt_pkg.cpython-312-x86_64-linux-gnu.so | grep '_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm'
                 U _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm@APTPKG_6.0


If we pass -lstdc++ -lapt-pkg we get the GLIBCXX one:

x86_64-linux-gnu-g++ -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 build/temp.linux-x86_64-cpython-312/python/acquire-item.o build/temp.linux-x86_64-cpython-312/python/acquire.o build/temp.linux-x86_64-cpython-312/python/apt_pkgmodule.o build/temp.linux-x86_64-cpython-312/python/cache.o build/temp.linux-x86_64-cpython-312/python/cachegroup.o build/temp.linux-x86_64-cpython-312/python/cdrom.o build/temp.linux-x86_64-cpython-312/python/configuration.o build/temp.linux-x86_64-cpython-312/python/depcache.o build/temp.linux-x86_64-cpython-312/python/generic.o build/temp.linux-x86_64-cpython-312/python/hashes.o build/temp.linux-x86_64-cpython-312/python/hashstring.o build/temp.linux-x86_64-cpython-312/python/hashstringlist.o build/temp.linux-x86_64-cpython-312/python/indexfile.o build/temp.linux-x86_64-cpython-312/python/lock.o build/temp.linux-x86_64-cpython-312/python/metaindex.o build/temp.linux-x86_64-cpython-312/python/orderlist.o build/temp.linux-x86_64-cpython-312/python/pkgmanager.o build/temp.linux-x86_64-cpython-312/python/pkgmanagerprogress.o build/temp.linux-x86_64-cpython-312/python/pkgrecords.o build/temp.linux-x86_64-cpython-312/python/pkgsrcrecords.o build/temp.linux-x86_64-cpython-312/python/policy.o build/temp.linux-x86_64-cpython-312/python/progress.o build/temp.linux-x86_64-cpython-312/python/python-apt-helpers.o build/temp.linux-x86_64-cpython-312/python/sourcelist.o build/temp.linux-x86_64-cpython-312/python/string.o build/temp.linux-x86_64-cpython-312/python/tag.o -L/usr/lib/x86_64-linux-gnu -lstdc++ -lapt-pkg -o build/lib.linux-x86_64-cpython-312/apt_pkg.cpython-312-x86_64-linux-gnu.so
root@79f30361847b:/build/python-apt# nm -D  build/lib.linux-x86_64-cpython-312/apt_pkg.cpython-312-x86_64-linux-gnu.so | grep '_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm'
                 U _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm@GLIBCXX_3.4.21

My expectation would have been for ld as invoked by g++ to always pass
-lstdc++ first such that the symbol resolves to the libstdc++ one.