Dear Maintainer,

for reference, #1125030 is the bug report on apt-listbugs that led to
this report.

reportbug/querybts are also affected by this.  Querybts sometimes
reports "there's no such bug" but succeeds on a second attempt, due to
this problem.

Thanks for looking into this.

On Sat, 24 Jan 2026, Jürgen A. Erhard wrote:n

Hi all, thanks for reporting this.

I looked at the logs, and as near as I can tell, there has been exactly
one 503 error to soap.cgi since January 10th across all three hosts.

I suspect that what's happening here is we're triggering a gateway
timeout and the fastly gateway is returning a 503 while the soap.cgi
goes on to complete the request. This probably only fails when you
happen to hit an underlying host which is already under high load (or
a request that takes longer to load).

Can you show the timing of the requests that fail in addition to the
logs?


1: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1125030#42

Hi Don, thanks for your prompt investigation!
[...]

Is the following enough?


    $ date -R -u ; apt-listbugs list apt ; date -R -u
    Sun, 25 Jan 2026 12:06:16 +0000
    Retrieving bug reports... 0% Fail
    E: HTTP GET failed (503: All backends failed or unhealthy)
    Retry downloading bug information? [Y/n] n
    Continue the installation anyway? [y/N]
    E: Exiting with error
    Sun, 25 Jan 2026 12:06:20 +0000

    $ date -R -u ; apt-listbugs list apt ; date -R -u
    Sun, 25 Jan 2026 12:11:08 +0000
    Retrieving bug reports... 0% Fail
    E: HTTP GET failed (503: Service Unavailable)
    Retry downloading bug information? [Y/n] n
    Continue the installation anyway? [y/N]
    E: Exiting with error
    Sun, 25 Jan 2026 12:11:11 +0000


It is almost always failing today...

On Sun, 25 Jan 2026, Francesco Poli wrote:n

Yeah; that's enough.

Given that these requests are only taking 3-4 seconds, it's definitely
not hitting a timeout. I'm not seeing any corresponding log records with
a 503 error on the backend (there are multiple 200 responses in this
time which look like they could correspond).

If you can set a specific user-agent string, I can check to see whether
I'm seeing the exact request that you sent in the backend logs. [The
observability of the current debbugs instance is pretty archaic, so it's
hard to connect an exact request to an exact set of logs.]

Thanks!

[...]

If I haven't messed things up, I should have set the user-agent to

"invernomuto (hopefully unique agent) (2.9.0, ruby 3.3.8 (2025-04-09))"

and I performed the following try:

    $ date -R -u ; apt-listbugs list apt ; date -R -u
    Sun, 25 Jan 2026 22:00:32 +0000
    Retrieving bug reports... 0% Fail
    E: HTTP GET failed (503: Service Unavailable)
    Retry downloading bug information? [Y/n] n
    Continue the installation anyway? [y/N]
    E: Exiting with error
    Sun, 25 Jan 2026 22:00:35 +0000


I hope this may help in pinpointing the issue.
issue!




Note to self: in order to modify the user-agent string, I temporarily
changed line 145
of /usr/share/rubygems-integration/all/gems/soap4r-ruby1.9-2.0.5/lib/soap/streamHandler.rb
(package 'ruby-soap4r') from

    @client = Client.new(nil, "SOAP4R/#{ Version }")

to

    @client = Client.new(nil, "invernomuto (hopefully unique agent)")

On Sun, 25 Jan 2026, Francesco Poli wrote:n

Hrm; I'm not seeing that user agent.

I suspect fastly is intercepting the request and returning the 503 even
before it hits bugs.debian.org.

I suspect if you switch to bugs-master.debian.org or
bugs-buxtehude.debian.org you won't see any 503 errors, but that's not a
configuration we want running in production.

Let me get someone from the admin team to see what's going on there.

hi Don,

I tried the 2 suggested URL's without any success :

jim@debian:~$ date; apt-listbugs -u
https://bugs-buxtehude.debian.org/cgi-bin/soap.cgi list apt
mar 27 jan 2026 15:59:56 CET
Récupération des rapports de bogue… Fait
Analyse des informations Trouvé/Corrigé… Fait
bogues de gravité serious sur apt (→ ) <En attente de traitement>
  b1 - #1078608 - apt update silently leaves old index data
bogues de gravité grave sur apt (→ ) <Résolus dans une version donnée>
/usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:920:in `+': incompatible
character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)
	from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:920:in `block (4
levels) in display_bugs'
	from /usr/lib/ruby/vendor_ruby/aptlistbugs/debian/bug.rb:69:in `block
in each_by_category'
	from /usr/lib/ruby/vendor_ruby/aptlistbugs/debian/bug.rb:68:in `each'
	from /usr/lib/ruby/vendor_ruby/aptlistbugs/debian/bug.rb:68:in
`each_by_category'
	from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:899:in `block (3
levels) in display_bugs'
	from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:895:in `each'
	from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:895:in `block (2
levels) in display_bugs'
	from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:894:in `each'
	from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:894:in `block in
display_bugs'
	from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:893:in `each'
	from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:893:in `display_bugs'
	from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:538:in `view'
	from /usr/bin/apt-listbugs:254:in `<main>'

jim@debian:~$ date; apt-listbugs -u
https://bugs-master.debian.org/cgi-bin/soap.cgi list apt
mar 27 jan 2026 16:00:15 CET
Récupération des rapports de bogue… 0% Échec
Err : HTTP GET a échoué
Réessayer de télécharger les informations du bogue ? [Y/n] ^CInterrupted

jim@debian:~$ date; apt-listbugs -u
https://bugs-master.debian.org/cgi-bin/soap.cgi list apt
mar 27 jan 2026 16:00:25 CET
Récupération des rapports de bogue… 0% Échec
Err : HTTP GET a échoué
Réessayer de télécharger les informations du bogue ? [Y/n] ^CInterrupted

jim@debian:~$ date; apt-listbugs -u
https://bugs-master.debian.org/cgi-bin/soap.cgi list apt
mar 27 jan 2026 16:00:35 CET
Récupération des rapports de bogue… Fait
Analyse des informations Trouvé/Corrigé… Fait
bogues de gravité serious sur apt (→ ) <En attente de traitement>
  b1 - #1078608 - apt update silently leaves old index data
bogues de gravité grave sur apt (→ ) <Résolus dans une version donnée>
/usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:920:in `+': incompatible
character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)
	from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:920:in `block (4
levels) in display_bugs'
	from /usr/lib/ruby/vendor_ruby/aptlistbugs/debian/bug.rb:69:in `block
in each_by_category'
	from /usr/lib/ruby/vendor_ruby/aptlistbugs/debian/bug.rb:68:in `each'
	from /usr/lib/ruby/vendor_ruby/aptlistbugs/debian/bug.rb:68:in
`each_by_category'
	from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:899:in `block (3
levels) in display_bugs'
	from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:895:in `each'
	from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:895:in `block (2
levels) in display_bugs'
	from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:894:in `each'
	from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:894:in `block in
display_bugs'
	from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:893:in `each'
	from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:893:in `display_bugs'
	from /usr/lib/ruby/vendor_ruby/aptlistbugs/logic.rb:538:in `view'
	from /usr/bin/apt-listbugs:254:in `<main>'

Regards,

Hello all

We are experiencing this issue (503 Service Unavailable) too, every time
the website is built (6 times a day), when we try to build wnpp data.

Here is the relevant part of our code:

my $soap =
SOAP::Lite->uri('Debbugs/SOAP')->proxy('https://bugs.debian.org/cgi-bin/soap.cgi')
        or die "Couldn't make connection to SOAP interface: $@";
my $bugs = $soap->get_bugs(package=>'wnpp')->result
        or die "Failed to get the list of bugs for package wnpp";
my $status = {};
my $count = 0;
my $total = @$bugs;
while (my @slice = splice(@$bugs, 0, 1000)) {
     $count = $count + 1;
     my $tmp = $soap->get_status(@slice)->result()
         or die "Failed to get status for bunch $count (of 1000 bugs),
total bugs to process: $total)";
     %$status = (%$status, %$tmp);
}

The complete script is here:
https://salsa.debian.org/webmaster-team/webwml/-/blob/master/english/devel/wnpp/wnpp.pl#L41
(anchor leads to the relevant code)

and here the error we get in the logs:
/srv/www.debian.org/cron/log/wml_run.log-make -C wnpp install
/srv/www.debian.org/cron/log/wml_run.log-make[3]: Entering directory
'/srv/www.debian.org/webwml/english/devel/wnpp'
/srv/www.debian.org/cron/log/wml_run.log-wml -q -D CUR_YEAR=2026 -o
UNDEFuEN:../../../english/devel/wnpp/wnpp.data@g+w -W7,-Squotes
../../../english/devel/wnpp/wnpp.pl
/srv/www.debian.org/cron/log/wml_run.log:ePerl:Error: Perl runtime error
(interpreter rc=2)
/srv/www.debian.org/cron/log/wml_run.log-
/srv/www.debian.org/cron/log/wml_run.log----- Contents of STDERR
channel: ---------
/srv/www.debian.org/cron/log/wml_run.log-503 Service Unavailable at
/tmp/8QkhgWG5xO/wml.tmp1 line 146.
/srv/www.debian.org/cron/log/wml_run.log-------------------------------------------
/srv/www.debian.org/cron/log/wml_run.log:** WML:Break: Error in Pass 3
(rc=1).
/srv/www.debian.org/cron/log/wml_run.log-Died at
/usr/share/wml/TheWML/Frontends/Wml/Runner.pm line 402.
/srv/www.debian.org/cron/log/wml_run.log-
TheWML::Frontends::Wml::Runner::_run_pass(TheWML::Frontends::Wml::Runner=HASH(0x565028e43830),
3, SCALAR(0x5650293d9af8), REF(0x5650293d9ac8), REF(0x5650293d9ae0))
called at /usr/share/wml/TheWML/Frontends/Wml/Runner.pm line 440
/srv/www.debian.org/cron/log/wml_run.log-
TheWML::Frontends::Wml::Runner::_passes_loop(TheWML::Frontends::Wml::Runner=HASH(0x565028e43830))
called at /usr/share/wml/TheWML/Frontends/Wml/Runner.pm line 727
/srv/www.debian.org/cron/log/wml_run.log-
TheWML::Frontends::Wml::Runner::_output_and_cleanup(TheWML::Frontends::Wml::Runner=HASH(0x565028e43830))
called at /usr/share/wml/TheWML/Frontends/Wml/Runner.pm line 932
/srv/www.debian.org/cron/log/wml_run.log-
TheWML::Frontends::Wml::Runner::run_with_ARGV(TheWML::Frontends::Wml::Runner=HASH(0x565028e43830),
HASH(0x5650291c8498)) called at /usr/bin/wml line 47
/srv/www.debian.org/cron/log/wml_run.log:make[3]: *** [Makefile:34:
../../../english/devel/wnpp/wnpp.data] Error 2
/srv/www.debian.org/cron/log/wml_run.log-make[3]: Leaving directory
'/srv/www.debian.org/webwml/english/devel/wnpp'
/srv/www.debian.org/cron/log/wml_run.log:make[2]:
[../../Makefile.common:84: wnpp-install] Error 2 (ignored)

(The error appears several times in each build since all languages after
English also try to connect).
The last complete log is always found in
https://www-master.debian.org/build-logs/webwml/wml_run.log

I can reproduce the error if I log into www-master.debian.org, go to
the  /srv/www.debian.org/webwml/english/devel/wnpp folder, and run sudo
-u debwww make

If you want me to change the code to make any test, just tell, but I
cannot create a test myself (not enough programming skills).

Thanks for your help!

Kind regards,

Laura Arjona Reina

https://wiki.debian.org/LauraArjona

Hi,

Laura Arjona Reina <larjona@debian.org> wrote (Tue, 27 Jan 2026 21:30:32 +0100):

Moreover, I just realized today that the spiderbts job from
https://i18n.debian.org/spiderbts/
is also affected (running 1 time per hour ! ), with

503 Service Unavailable at /srv/i18n.debian.org//dl10n/git/lib/Debian/L10n/BTS.pm line 115.

See
https://salsa.debian.org/l10n-team/dl10n/-/blame/master/lib/Debian/L10n/BTS.pm#L115


Aka, the spiderbts service is currently effectively down!


Holger

[...]

I tried a few times some hours ago and (if I recall correctly) I saw
one 503 from bugs-master.debian.org, one successful response from
bugs-master.debian.org, and some successful responses from
bugs-buxtehude.debian.org (no 503 from bugs-buxtehude.debian.org)

I know this is kind of vague, sorry about that, but I was in a hurry.

I am retrying now, but I cannot reproduce the issue anymore.
Just a calmer time, or has anything been fixed on fastly?

Have you managed to get someone from the admin team who fixed the
issue, perhaps?!?

about one hour ago:

Error retrieving bug reports from the server with the following error message:
E: Failed to open TCP connection to bugs.debian.org:80 (Connection refused - connect(2) for 209.87.16.81:80)

And this still fails:

joooj:~> curl http://bugs.debian.org
curl: (7) Failed to connect to bugs.debian.org port 80 after 270 ms: Couldn't connect to server

But I'm wondering why apt-listbugs uses port 80.

It no longer does (unless you force it to do so), since it switched to
https by default.
But that happened in apt-listbugs/0.1.42, while bookworm has
apt-listbugs/0.1.40 ...

but the following new one has just occurred:

Error retrieving bug reports from the server with the following error message:
E: HTTPClient::KeepAliveDisconnected:
It could be because your network is down, or because of broken proxy servers, or the BTS server itself is down. Check network configuration and try again

That should work now.

It appears that up to (and including) bookworm, TLS support was
resisted due to concerns around OpenSSL licensing. From trixie onwards
HTTPS is used.

Regards,

Adam

There was a brief issue while port 80 support was being added.
Everything should be working again.

Regards,

Adam

[...]

The situation looks definitely improved, although I was seeing the
following about one hour ago:

    $ apt-listbugs list grub-efi-amd64
    Retrieving bug reports... 0% Fail
    E: HTTP GET failed (503: Backend fetch failed)
    Retry downloading bug information? [Y/n]
    One package at a time? [Y/n] n
    One bug report at a time? [Y/n] n
     Fail
    E: HTTP GET failed (503: Backend fetch failed)
    Retry downloading bug information? [Y/n]
    One package at a time? [Y/n] n
    One bug report at a time? [Y/n] n
    Retrieving bug reports... Done
    Parsing Found/Fixed information... Done
    critical bugs of grub-efi-amd64 (→ ) <Outstanding>
     b1 - #1102160 - upgrade-reports: Bookworm to Trixie [amd64][EFI] initramfs unpacking failed invalid magic at start of compressed archive
     b2 - #913576 - Update to grub-efi-amd64 2.02~beta3-5 results in "error: efibootmgr failed to register the boot entry: Operation not permitted."
     b3 - #913916 - grub-efi-amd64: UEFI boot option removed after update to grub2 2.02~beta3-5+deb9u1
     b4 - #984520 - 'error: symbol "grub_register_command_lockdown" not found' and then lightdm fails to start
    Summary:
     grub-efi-amd64(4 bugs)


But now, I am not seeing any errors:

    $ apt-listbugs list grub-efi-amd64
    Retrieving bug reports... Done
    Parsing Found/Fixed information... Done
    critical bugs of grub-efi-amd64 (→ ) <Outstanding>
     b1 - #1102160 - upgrade-reports: Bookworm to Trixie [amd64][EFI] initramfs unpacking failed invalid magic at start of compressed archive
     b2 - #913576 - Update to grub-efi-amd64 2.02~beta3-5 results in "error: efibootmgr failed to register the boot entry: Operation not permitted."
     b3 - #913916 - grub-efi-amd64: UEFI boot option removed after update to grub2 2.02~beta3-5+deb9u1
     b4 - #984520 - 'error: symbol "grub_register_command_lockdown" not found' and then lightdm fails to start
    Summary:
     grub-efi-amd64(4 bugs)

Exactly.

Bye Adam, and thanks to you and to the other involved people for
fixing/improving the situation!
Your time and dedication is much appreciated.

Hello again

An update from the website (build jobs of www.debian.org): we're getting
successful builds most of the time now.

Today we have gotten a similar error (the error message is slightly
different) only once:

503 Backend fetch failed

during the job that started at
Thu Jan 29 15:24:01 UTC 2026
and finished at
Thu Jan 29 16:33:22 UTC 2026

The rest of the builds (before and after, so far) went well.
Thanks a lot to the people involved in handling the issue.
If it gets worse again I will follow up writing to the bug.

Kind regards

This is apparently not fixed yet as I've just got

Retrieving bug reports... 0% Fail
E: HTTP GET failed
Retry downloading bug information? [Y/n]

Error retrieving information on existing bug reports from the BTS. The following error was detected:
Unable to connect to Debian BTS (error: "<HTTPError 503: 'Backend fetch failed'>");

And this is not just SOAP. When looking at a bug page:

Error 503 Backend fetch failed

Backend fetch failed
Guru Meditation:

XID: 2876932920

Varnish cache server