Dear apt team,

thanks for your work, it's really apreciated :-)

Today, I was doing an 'aptitude upgrade' in a server running stretch testing,
which was a bit outdated.

This is the result of the 'aptitude upgrade' command. As you can see, when
aptitude tries to deal with libapt-pkg5.0, some kind of chiken-egg problem
happens:

(BTW, running 'aptitude upgrade' again resolves the problem)

===== 8< =====
sudo aptitude upgrade
Resolving dependencies...
The following packages will be upgraded:
  apt apt-utils aptitude-doc-en bash bind9-host binutils busybox dnsutils file geoip-database git git-man host init init-system-helpers initramfs-tools initramfs-tools-core iso-codes libapt-inst2.0
  libapt-pkg5.0 libavahi-client3 libavahi-common-data libavahi-common3 libbind9-140 libc-bin libc-l10n libc6 libcurl3-gnutls libdns-export162 libdns162 libfreetype6 libgmime-2.6-0 libisc-export160
  libisc160 libisccc140 libisccfg140 liblwres141 libmagic-mgc libmagic1 libsystemd0 libtirpc1 libudev1 locales multiarch-support python-apt python-apt-common python-magic python3-apt rpcbind systemd
  systemd-sysv udev vim-common vim-tiny xxd zsh
The following packages are RECOMMENDED but will NOT be installed:
  libpam-systemd
56 packages upgraded, 0 newly installed, 0 to remove and 40 not upgraded.
Need to get 38.8 MB of archives. After unpacking 658 kB will be used.
Do you want to continue? [Y/n/?] Y
Get: 1 http://ftp.cica.es/debian/ testing/main bash amd64 4.4-4+b2 [1427 kB]
Get: 2 http://ftp.cica.es/debian/ testing/main libc6 amd64 2.24-10 [2691 kB]
Get: 3 http://ftp.cica.es/debian/ testing/main libapt-pkg5.0 amd64 1.4.1 [916 kB]
Get: 4 http://ftp.cica.es/debian/ testing/main libapt-inst2.0 amd64 1.4.1 [192 kB]
Get: 5 http://ftp.cica.es/debian/ testing/main apt amd64 1.4.1 [1231 kB]
Get: 6 http://ftp.cica.es/debian/ testing/main apt-utils amd64 1.4.1 [410 kB]
Get: 7 http://ftp.cica.es/debian/ testing/main init-system-helpers all 1.48 [44.0 kB]
Get: 8 http://ftp.cica.es/debian/ testing/main libtirpc1 amd64 0.2.5-1.2 [80.6 kB]
Get: 9 http://ftp.cica.es/debian/ testing/main libudev1 amd64 232-23 [123 kB]
Get: 10 http://ftp.cica.es/debian/ testing/main udev amd64 232-23 [1110 kB]
Get: 11 http://ftp.cica.es/debian/ testing/main systemd amd64 232-23 [2461 kB]
Get: 12 http://ftp.cica.es/debian/ testing/main libsystemd0 amd64 232-23 [278 kB]
Get: 13 http://ftp.cica.es/debian/ testing/main rpcbind amd64 0.2.3-0.6 [46.0 kB]
Get: 14 http://ftp.cica.es/debian/ testing/main libc-bin amd64 2.24-10 [772 kB]
Get: 15 http://ftp.cica.es/debian/ testing/main init amd64 1.48 [12.0 kB]
Get: 16 http://ftp.cica.es/debian/ testing/main busybox amd64 1:1.22.0-19+b3 [405 kB]
Get: 17 http://ftp.cica.es/debian/ testing/main initramfs-tools all 0.130 [66.0 kB]
Get: 18 http://ftp.cica.es/debian/ testing/main initramfs-tools-core all 0.130 [97.0 kB]
Get: 19 http://ftp.cica.es/debian/ testing/main systemd-sysv amd64 232-23 [79.7 kB]
Get: 20 http://ftp.cica.es/debian/ testing/main git-man all 1:2.11.0-3 [1431 kB]
Get: 21 http://ftp.cica.es/debian/ testing/main libcurl3-gnutls amd64 7.52.1-5 [289 kB]
Get: 22 http://ftp.cica.es/debian/ testing/main git amd64 1:2.11.0-3 [4151 kB]
Get: 23 http://ftp.cica.es/debian/ testing/main zsh amd64 5.3.1-4+b1 [816 kB]
Get: 24 http://ftp.cica.es/debian/ testing/main multiarch-support amd64 2.24-10 [199 kB]
Get: 25 http://ftp.cica.es/debian/ testing/main xxd amd64 2:8.0.0197-4 [131 kB]
Get: 26 http://ftp.cica.es/debian/ testing/main vim-tiny amd64 2:8.0.0197-4 [446 kB]
Get: 27 http://ftp.cica.es/debian/ testing/main vim-common all 2:8.0.0197-4 [159 kB]
Get: 28 http://ftp.cica.es/debian/ testing/main dnsutils amd64 1:9.10.3.dfsg.P4-12.2 [283 kB]
Get: 29 http://ftp.cica.es/debian/ testing/main bind9-host amd64 1:9.10.3.dfsg.P4-12.2 [230 kB]
Get: 30 http://ftp.cica.es/debian/ testing/main libisc160 amd64 1:9.10.3.dfsg.P4-12.2 [398 kB]
Get: 31 http://ftp.cica.es/debian/ testing/main libdns162 amd64 1:9.10.3.dfsg.P4-12.2 [1076 kB]
Get: 32 http://ftp.cica.es/debian/ testing/main libisccc140 amd64 1:9.10.3.dfsg.P4-12.2 [198 kB]
Get: 33 http://ftp.cica.es/debian/ testing/main libisccfg140 amd64 1:9.10.3.dfsg.P4-12.2 [222 kB]
Get: 34 http://ftp.cica.es/debian/ testing/main liblwres141 amd64 1:9.10.3.dfsg.P4-12.2 [213 kB]
Get: 35 http://ftp.cica.es/debian/ testing/main host all 1:9.10.3.dfsg.P4-12.2 [185 kB]
Get: 36 http://ftp.cica.es/debian/ testing/main libbind9-140 amd64 1:9.10.3.dfsg.P4-12.2 [205 kB]
Get: 37 http://ftp.cica.es/debian/ testing/main file amd64 1:5.30-1 [63.9 kB]
Get: 38 http://ftp.cica.es/debian/ testing/main python-magic all 1:5.30-1 [48.1 kB]
Get: 39 http://ftp.cica.es/debian/ testing/main libmagic1 amd64 1:5.30-1 [111 kB]
Get: 40 http://ftp.cica.es/debian/ testing/main libmagic-mgc amd64 1:5.30-1 [222 kB]
Get: 41 http://ftp.cica.es/debian/ testing/main libc-l10n all 2.24-10 [819 kB]
Get: 42 http://ftp.cica.es/debian/ testing/main locales all 2.24-10 [3287 kB]
Get: 43 http://ftp.cica.es/debian/ testing/main aptitude-doc-en all 0.8.7-1 [434 kB]
Get: 44 http://ftp.cica.es/debian/ testing/main binutils amd64 2.28-5 [3770 kB]
Get: 45 http://ftp.cica.es/debian/ testing/main geoip-database all 20170418-1 [2030 kB]
Get: 46 http://ftp.cica.es/debian/ testing/main iso-codes all 3.75-1 [2389 kB]
Get: 47 http://ftp.cica.es/debian/ testing/main libavahi-common-data amd64 0.6.32-2 [118 kB]
Get: 48 http://ftp.cica.es/debian/ testing/main libavahi-common3 amd64 0.6.32-2 [52.0 kB]
Get: 49 http://ftp.cica.es/debian/ testing/main libavahi-client3 amd64 0.6.32-2 [55.3 kB]
Get: 50 http://ftp.cica.es/debian/ testing/main libisc-export160 amd64 1:9.10.3.dfsg.P4-12.2 [334 kB]
Get: 51 http://ftp.cica.es/debian/ testing/main libdns-export162 amd64 1:9.10.3.dfsg.P4-12.2 [851 kB]
Get: 52 http://ftp.cica.es/debian/ testing/main libfreetype6 amd64 2.6.3-3.2 [438 kB]
Get: 53 http://ftp.cica.es/debian/ testing/main libgmime-2.6-0 amd64 2.6.22+dfsg2-1 [232 kB]
Get: 54 http://ftp.cica.es/debian/ testing/main python-apt-common all 1.4.0~beta3 [93.0 kB]
Get: 55 http://ftp.cica.es/debian/ testing/main python3-apt amd64 1.4.0~beta3 [170 kB]
Get: 56 http://ftp.cica.es/debian/ testing/main python-apt amd64 1.4.0~beta3 [172 kB]
Fetched 38.8 MB in 17s (2168 kB/s)
Reading changelogs... Done
apt-listchanges: Mailing root: apt-listchanges: news for debliquid
Extracting templates from packages: 100%
Preconfiguring packages ...
(Reading database ... 60361 files and directories currently installed.)
Preparing to unpack .../bash_4.4-4+b2_amd64.deb ...
Unpacking bash (4.4-4+b2) over (4.4-4+b1) ...
Processing triggers for install-info (6.3.0.dfsg.1-1+b2) ...
Processing triggers for man-db (2.7.6.1-2) ...
Setting up bash (4.4-4+b2) ...
update-alternatives: using /usr/share/man/man7/bash-builtins.7.gz to provide /usr/share/man/man7/builtins.7.gz (builtins.7.gz) in auto mode
(Reading database ... 60361 files and directories currently installed.)
Preparing to unpack .../libc6_2.24-10_amd64.deb ...
Unpacking libc6:amd64 (2.24-10) over (2.24-9) ...
Setting up libc6:amd64 (2.24-10) ...
Processing triggers for libc-bin (2.24-9) ...
(Reading database ... 60361 files and directories currently installed.)
Preparing to unpack .../libapt-pkg5.0_1.4.1_amd64.deb ...
Unpacking libapt-pkg5.0:amd64 (1.4.1) over (1.4) ...
Processing triggers for libc-bin (2.24-9) ...
dpkg: error: dpkg status database is locked by another process
E: Sub-process /usr/bin/dpkg returned an error code (2)
Failed to perform requested operation on package.  Trying to recover:
dpkg: error: dpkg status database is locked by another process
E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
E: Could not regain the system lock!  (Perhaps another apt or dpkg is running?)
E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?

Current status: 93 updates [-3].
W: Could not lock the cache file; this usually means that dpkg or another apt tool is already installing packages.  Opening in read-only mode; any changes you make to the states of packages will NOT be preserved!
===== 8< =====

Control: reassign -1 aptitude
[...]

Thanks for the bug report. There is a known race condition between
dpkg and apt tools, but I think we mostly worked around that for
now in apt.

The real fix for this is:

https://lists.debian.org/debian-dpkg/2017/01/msg00044.html

But that requires changes basically everywhere. Once that is solved
on the dpkg side, we can fix stuff elsewhere. That is tracked in
Bug #850417.

Reassigning to aptitude for now, as maybe aptitude does not do
its best to reduce the race as much as possible (keep the lock
until just before DoInstall(), and lock again once that is
done).

If it does, I guess we should probably reassign to dpkg and
forcemerge with 850417.

2017-05-16 09:39 Julian Andres Klode:

aptitude doesn't do anything special for this, and there are previous
bugs about it (copying it for future triage).

Even if maybe in practice it's better to do what you propose, I always
thought that it was (theoretically) the wrong way to deal with locks,
because in the interim between unlock in dpkg and the lock-again in
apt*-like tools another competing process can eat the cake.

But yeah, maybe by avoiding to do the perfect solution the end result is
worse.

In any case, I didn't have to do much time to deal with this, but even
if I did, I think that it's a potentially problematic change that it's
not good to implement in the "deep freeze".

So now, any resolution it'll have to wait until after the release,
sorry.


Cheers.

-- 

Hello,

How are you doing today? I sent you an email yesterday, did you receive it?
It is a very important message, anyway reply back to confirm that you
already got my message to enable me to give you more details..

Best Regards.
Mrs. Ameena Essa