Dear all
Since several days, the script wnpp.pl fails intermittently when trying to connect to bugs.debian.org, and when it fails, we receive the following messages in the webwml build log:


[...]
make -C wnpp install
make[3]: Entering directory '/srv/www.debian.org/webwml/english/devel/wnpp'
wml -q -D CUR_YEAR=2022 -o UNDEFuEN:../../../english/devel/wnpp/wnpp.data@g+w   -W7,-Squotes ../../../english/devel/wnpp/wnpp.pl
ePerl:Error: Perl runtime error (interpreter rc=2)
---- Contents of STDERR channel: ---------
500 Can't connect to bugs.debian.org:443 (Connection refused) at /tmp/wml.12011.tmp1 line 147.
------------------------------------------
** WML:Break: Error in Pass 3 (rc=1).
Died at /usr/share/wml/TheWML/Frontends/Wml/Runner.pm line 403.
	TheWML::Frontends::Wml::Runner::_run_pass(TheWML::Frontends::Wml::Runner=HASH(0x55f9849f2458), 3, SCALAR(0x55f984f718d8), SCALAR(0x55f984f718a8), SCALAR(0x55f984f718c0)) called at /usr/share/wml/TheWML/Frontends/Wml/Runner.pm line 441
	TheWML::Frontends::Wml::Runner::_passes_loop(TheWML::Frontends::Wml::Runner=HASH(0x55f9849f2458)) called at /usr/share/wml/TheWML/Frontends/Wml/Runner.pm line 726
	TheWML::Frontends::Wml::Runner::_output_and_cleanup(TheWML::Frontends::Wml::Runner=HASH(0x55f9849f2458)) called at /usr/share/wml/TheWML/Frontends/Wml/Runner.pm line 930
	TheWML::Frontends::Wml::Runner::run_with_ARGV(TheWML::Frontends::Wml::Runner=HASH(0x55f9849f2458), HASH(0x55f984b252b8)) called at /usr/bin/wml line 47
make[3]: *** [Makefile:34: ../../../english/devel/wnpp/wnpp.data] Error 2
make[3]: Leaving directory '/srv/www.debian.org/webwml/english/devel/wnpp'
make[2]: [../../Makefile.common:84: wnpp-install] Error 2 (ignored)
make[2]: Leaving directory '/srv/www.debian.org/webwml/english/devel'

[...]

Since the line refers to a temp file, I'm not 100% sure about which is the command that causes the failure, but the code in wnpp.pl that uses bugs.debian.org I think is restricted to these lines (initial connection, and retrieval of bug info to build the wnpp.data local database):

https://salsa.debian.org/webmaster-team/webwml/-/blob/master/english/devel/wnpp/wnpp.pl#L41 (L41 to 48)

my $soap = SOAP::Lite->uri('Debbugs/SOAP')->proxy('https://bugs.debian.org/cgi-bin/soap.cgi')
        or die "Couldn't make connection to SOAP interface: $@";
my $bugs = $soap->get_bugs(package=>'wnpp')->result;
my $status = {};
while (my @slice = splice(@$bugs, 0, 500)) {
     my $tmp = $soap->get_status(@slice)->result() or die;
     %$status = (%$status, %$tmp);
}

Some people commented the issue in the #debian-admin IRC channel, for the case there was some issue in the bugs.d.o servers or some configuration change, but it seems the failures are related to the connections being blocked by the http rate-limiting. It's not clear, however, why these issues are appearing now (nothing in the script changed in the recent times, and the amount of bugs is not drastically bigger than weeks before). In addition to this, some other days the script is tolerated well (for example, we had failures on 2022/02/18 and 2022/02/22, but today 2022/02/23 the builds are finishing successfully).

Looking at the code, I see we're processing bugs in groups of 500 bugs. I'm not sure if we could try to reduce the number of the bugs in the group and introduce some "sleep" time between queries, or it's better to do the opposite, increase the number of bugs in a group to reduce the total number of queries.

Other approach could be to rewrite the wnpp.pl script so it uses queries against UDD instead of the SOAP CGI interface of bugs.debian.org

I don't know which approach is better, and also my Perl skills are not enough to propose a patch myself for none of this two options. If anybody can and wants to work on this, it's very appreciated.

I'll point the Debbugs team to this bug one it is created for the case they changed any configuration in their deployment that caused the connection failures, or for the case they have any advice about the better way to connect or retrieve the information we need to create the WNPP-related web pages.

Kind regards,

Laura Arjona Reina
https://wiki.debian.org/LauraArjona

Hello
I have added some lines to the wnpp.pl script (see commit below) so when it fails, it shows the point of the process when the connection failed.
Kind regards,

Hello all
The code I added to show more info when it fails didn't help, we get the same messages in the build logs.

I have tried to use groups of 300 bugs instead of 500, and also groups of 1000 bugs, but it seems it makes no difference.

I have looked at the date of the wnpp.data file in www-master.d.o and it's from today 03:29, so I guess the script runs successfully the first time of the day, and then it fails on the later builds.

Looking at the Makefile, that tries to generate the wnpp.data for English in every build, and also deletes the file on error, I'm thinking about moving the code generating the wnpp.data file to the "lessoften" cron scripts so it runs once a day, and let the webwml repo to just use the wnpp.data file of that day.

Opinions?

Kind regards