#973581 apt: Occasional pipelined requests failing against localhost nginx mirror

Package:
apt
Source:
apt
Description:
commandline package manager
Submitter:
Stefano Rivera
Date:
2022-02-09 17:51:02 UTC
Severity:
normal
#973581#5
Date:
2020-11-02 04:55:36 UTC
From:
To:
I have a local Debian mirror on a buster machine that I occasionally
build packages on too. Pretty stock Debian nginx config.

Recently I noticed repeated sbuild failures during package installation,
due to connection resets. Can't reproduce with apt 2.1.8 but can with
fa375493c5a4ed9c10d4e5257ac82c6e687862d3 disabling retries.

Clearly pipelining messups are still present.

It's quite repeatable on an "apt build-dep chromium" in a sid chroot.
Usually 2 packages will fail to download.

I took a stab at finding the source of the resets, but it's nothing
obvious. I see nginx resetting the connection about 100MiB into it,
immediately after an APT GET request. Those kind of pcaps are too big to
share and not that useful. But I can give you access to experiment, if
desired. Prod me by private email / IRC.

It's entirely possible that I simply haven't built anything on that
machine since apt 2.1.9 that triggers this bug.

SR

#973581#10
Date:
2022-02-08 19:21:44 UTC
From:
To:
I hit this bug as well with our debian mirror, mirrors.wikimedia.org.
We also use nginx to serve our mirror and bullseye clients frequently
receive resets while pulling down packages. I tested reverting commit,
fa375493c5a4ed9c10d4e5257ac82c6e687862d3, and it does indeed stop the
connection resets. I also tested against apache2, but I was unable to
reproduce the issue, so it seems to be nginx specific. I am happy to provide
pcaps as well.

Here is the script I used to reproduce:

  #!/bin/bash

  set -o errexit

  if [[ $EUID -ne 0 ]]; then
    printf 'MUST BE ROOT\n' >&2
    exit 1
  fi

  sed -i 's/deb.debian.org/mirrors.wikimedia.org/' /etc/apt/sources.list
  apt-get update
  while true; do
    apt-get clean
    declare -a install_opts=(--yes --download-only --no-install-recommends)
    declare -a install_pkgs=(firefox-esr chromium)
    if ! apt-get install "${install_opts[@]}" "${install_pkgs[@]}"; then
      printf 'ERROR: We caught a fish!\n' >&2
      tput bel
      exit 1
    fi
  done

Here is how I built apt to test the revert:

  #!/bin/bash
  set -o errexit

  echo 'deb-src http://deb.debian.org/debian bullseye main'
  apt-get update
  apt-get install --yes git build-essential
  apt-get build-dep --yes apt
  git clone https://salsa.debian.org/apt-team/apt.git
  git config --global user.email "jhathaway@wikimedia.org"
  git config --global user.name "Jesse Hathaway"
  pushd apt
  git checkout 2.2.4
  git checkout -b butter
  git revert --no-edit fa375493c5a4ed9c10d4e5257ac82c6e687862d3
  dpkg-buildpackage -jauto -us -nc -b
  popd
  dpkg -i -- *.deb

I tested with buildah and this Dockerfile:

  FROM docker.io/debian:bullseye
  ENV LANG C.UTF-8
  COPY build /root/
  COPY resets /root/

#973581#15
Date:
2022-02-09 17:47:43 UTC
From:
To:
Using the same script I was able to reproduce the issue against the
following mirror as well:

- http://ftp.us.debian.org/debian/  Server: nginx/1.18.0
- http://debian.unnoba.edu.ar/debian/ Server: nginx/1.18.0
- http://mirrors.tuna.tsinghua.edu.cn/debian/ Server: nginx/1.18.0
- http://repo.ialab.dsu.edu/debian/ Server: nginx/1.18.0
- http://mirror.us.leaseweb.net/debian/ Server: nginx
- http://plug-mirror.rcac.purdue.edu/debian/  Server: nginx/1.14.2