https://udd.debian.org/udd-status.cgi is reporting that these Ubuntu
importers haven't run successfully since March 23, 2026:
- archive-ubuntu
- ubuntu-bugs
- ubuntu-upload-history

The ubuntu-upload-history log excerpt includes:

  urllib.error.HTTPError: HTTP Error 503: Service Unavailable
  ERROR:launchpad-ubuntu-changes:Error: Something broke badly.

I know that Launchpad has been having difficulty staying fully
available and fast in the past few weeks.

I'd expect for udd to retry imports later after failure but the "next
time" column also says March 23 so maybe these importers aren't
retrying any more?

Thank you,
Jeremy Bícha

Hi,

Thanks for the report.

I think that the reason is that ubuntu-upload-history got stuck, and as
a result, blocked the other importers because they are in the same
"queue" in the UDD orchestrator.

I added a timeout. Let's see if this fixes the issue.

Lucas

ubuntu_upload_history is still lagging behind (and Launchpad is
struggling today!), but the other Ubuntu jobs are up-to-date or close
enough to it. It's those other jobs that are powering the dashboards
that I use so the situation is much better for me now.

Thank you,
Jeremy Bícha

It looks like ullmann.debian.org (the machine behind udd.debian.org) is
backlisted by launchpad again.

ullmann:~$ curl -6 -kv https://launchpad.net/ubuntu/+archive/primary/+sourcepub/18280857/+files/changelog
* Host launchpad.net:443 was resolved.
* IPv6: 2620:2d:4000:1009::f3, 2620:2d:4000:1009::3ba
* IPv4: (none)
*   Trying [2620:2d:4000:1009::f3]:443...

^C

ullmann:~$ mtr -c 5 -r -6 -T -P 443 2620:2d:4000:1009::f3
Start: 2026-04-06T19:24:21+0000
HOST: ullmann                     Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2607:f8f0:614:1::2         0.0%     5    0.4   0.4   0.3   0.8   0.2
  2.|-- 2607:f8f0:610:4::302       0.0%     5    0.4   0.4   0.3   0.4   0.0
  3.|-- 2607:f8f0:610:4::305       0.0%     5    1.1   1.0   0.8   1.1   0.1
        2607:f8f0:610:4::304
  4.|-- 381-oran-cr1-ubc.vncv1.bc  0.0%     5    2.0   1.6   1.0   2.0   0.5
        2607:f8f0:610:4::1f1
  5.|-- 381-oran-cr1-ubc.vncv1.bc  0.0%     5  5105. 1634.   1.7 5105. 1990.1
  6.|-- 2607:f8f0:1400::16         0.0%     5  1029. 1025.   2.2 3062. 1249.4
  7.|-- 2607:f8f0:1400::16         0.0%     5  2050. 616.7   2.0 2050. 916.0
  8.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0
  9.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0
 10.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0
 11.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0
 12.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0
 13.|-- he-as6939.torix.ca         0.0%     5  3119. 1895.  52.1 4154. 1832.7
 14.|-- he-as6939.torix.ca         0.0%     5  2082. 1068.  51.3 2102. 1020.5
 15.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0
 16.|-- port-channel1.core1.lon6. 40.0%     5  127.7 809.4 127.7 1154. 590.4
 17.|-- swp9.il3-core2.canonical.  0.0%     5  1157. 741.4 126.6 2168. 913.9
        port-channel1.core1.lon9.he.net
        port-channel1.core1.lon6.he.net
        swp9.il3-core1.canonical.com
 18.|-- swp9.il3-core2.canonical.  0.0%     5  126.9 127.2 126.9 128.0   0.4
        swp9.il3-core1.canonical.com
        port-channel1.core1.lon6.he.net
 19.|-- port-channel1.core1.lon6.  0.0%     5  128.2 534.9 127.0 1157. 557.6
        vrrp.bond-core.il3-fw.canonical.com
 20.|-- swp9.il3-core2.canonical.  0.0%     5  3197. 1965. 127.4 4219. 1829.8
        swp9.il3-core1.canonical.com
 21.|-- vrrp.bond-core.il3-fw.can  0.0%     5  7249. 2167. 127.2 7249. 2965.1
 22.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0

ullmann.d.o is 2607:f8f0:614:1::1274:38

If some could reach out to the Canonical team in charge of this, it
would be great.

Lucas

Hey Lucas,

Thanks for flagging that, it had been indeed blocked again while they
were trying to cut some load issue due to IA bots/scapers.

Do you have an idea of the number of API requests/pages load UDD is
usually doing on a $period (day?), I wonder if the load is really that
high and if there is some client optimization that could be done in UDD...

Cheers,
Sébastien

Le 06/04/2026 à 21:26, Lucas Nussbaum a écrit :

Hi Sébastien,

AFAIK, there are two UDD processes that query launchpad.

One is ubuntu-upload-history, that should query Launchpad's API once per
upload (or twice, since the API redirects to launchpadlibrarian.net).

The other one is ubuntu-bugs, that refreshes the status of Ubuntu bugs
in UDD. That one runs on a regular basis (once per day currently).

The relevant code is
https://salsa.debian.org/qa/udd/-/blob/master/scripts/launchpad-ubuntu-changes?ref_type=heads
for the first script, and
https://salsa.debian.org/qa/udd/-/blob/master/udd/ubuntu_bugs_gatherer.py?ref_type=heads
for the second script.

The refresh frequency of ubuntu-bugs could probably be reduced if that
helps.

Lucas

Hey again Lucas,

Urg, no wonder UDD is getting blocked; the ubuntu_bugs_gatherer script
basically parse 163k launchpad pages, doing 10 parallel requesters
without any rate limiting and not slowing doing in case the service
struggles and start returning 429 or 503.

Ideally that would use launchpadlibAPI to use searchTasks() with a
modified_since= to limit the list to bugs that changed since the
previous run (if you have that information) or over a day (or a few days
to accomodate for flakyness).

Switching to use API instead of text pages parsing requires a bit of
work, but maybe a start for now would be to lower the number of parallel
workers (+ a small sleep(0.3) or something between calls + pause if the
service starts returning errors)?
The launchpad team confirmed that the number of queries/the daily
frequency isn't the problem but the "spike" is

Cheers,
Sébastien

Le 07/04/2026 à 16:24, Lucas Nussbaum a écrit :

Hi,

For now I reduced the parallelism (num_fetchers) from 10 to 2, which
should clearly help spread the load over time. Maybe let's see if that's
better? I could reduce it to 1 and add a delay, but long-running
importers are a bit painful for other reasons, so I'd rather avoid that.

Lucas

Hey again,

Le 07/04/2026 à 20:59, Lucas Nussbaum a écrit :
Yes, that sounds reasonable to me, thanks!

Sébastien

Hi,

It looks like ullmann.debian.org is blacklisted again.
`curl https://launchpad.net/ubuntu/+bugs-text` timeouts.

$ host ullmann.debian.org
ullmann.debian.org has address 209.87.16.38

Cheers,

Lucas

Hey Lucas,

Indeed, launchpad is still struggling with AI scrappers and similar and the
number of requests UDD is making has led to the IP being blocked again. It
is unblocked now, but I think we need figure out a way for it to not hammer
launchpad that hard on a regular basis (if my previous quick check is
correct is does read 160k+ pages every run). Was there any technical reason
to not use launchpad API (which would allow to filter on recent changes and
only process bugs that changed recently instead of hammering every
launchpad bug page at every run)?

Cheers,
Sébastien

Hi Sébastien,

Thanks!

Most of that code was written in 2008, so I don't remember the design
choices from back then. Maybe the launchpad API wasn't ready for that
back then.

Still, the main reason for re-importing all bugs every time is that it's
easier to ensure data correctness that way. Once you try to refresh only
things that change, you increase your chances of running into corner
cases...

I would welcome a patch to re-implement the code using the launchpad
API, but I'm unlikely to find time+motivation to work on it myself.

In the meantime, if that helps, I can reduce the number of parallel
workers from 2 to 1 -- I reorganized the process to split it into a
download phase, and an SQL INSERT phase, so if the download phase takes
several days, it's no longer a problem (previously the long running
transaction caused a problem because it prevented VACUUMing).

Lucas