#1141980 cloud-init fails to install packages with 'could not resolve' DNS error

#1141980#5
Date:
2026-07-13 07:51:35 UTC
From:
To:
I am using Debian Trixie generic cloud image (debian-13-amd64-20260706-2531.qcow2)
when setting up the basic configuration works, however anything requiring a network
connection, e.g. installing packages, fails with a 'could not resolve' error
when looking up the APT proxy. After the set-up fails I can log in to the VM
and perform an 'apt update' and 'apt upgrade' as normal.

It almost seems like the DNS configuration hasn't been fully applied yet by the time
cloud-init attempts to install packages. Although the logging indicates that the
network target has already been reached.

As this appears to be a timing (race?) issue, I am not sure how to diagnose it
futhter. Any suggestion are most welcome.

More specifically I am running the generic image as a QEMU/KVM VM on a Debian
Bookworm host. I am using the following command to start the VM:

virt-install --name cloudtest --vcpus vcpus=1,maxvcpus=2 --cpu host-passthrough \
--memory currentMemory=1024,memory=2048 --osinfo name=debiantesting --import \
--disk path=/dev/alpha/test-cloud,driver.discard=unmap \
--events on_poweroff=destroy,on_reboot=restart,on_crash=destroy \
--input type=keyboard --input type=mouse --input type=tablet \
--network network=guest,type=bridge,source=guest \
--cloud-init user-data=$(pwd)/user-data,meta-data=$(pwd)/meta-data,network-config=$(pwd)/network-config

The meta-data file is empty. The network-config files contains:

network:
  version: 2
  ethernets:
    net0:
      match:
        name: enp1s0
      dhcp4: false
      dhcp6: false
      accept-ra: false
      link-local: [ipv6]
      addresses:
        - [redacted].0.130/24
        - [redacted]:30::130/64
      routes:
        - to: default
          via: [redacted].0.254
          on-link: true
        - to: default
          via: [redacted]:30::fefe
          on-link: true
      nameservers:
        search: [mydomain.local]
        addresses:
          - [redacted].0.254

The user-data file contains:

#cloud-config
fqdn: "cloudtest.mydomain.local"
manage_etc_hosts: true
users:
  - name: "laurensb"
    gecos: "Laurens Blankers"
    ssh_authorized_keys:
      - "ssh-rsa [redacted]"
    sudo: "ALL=(ALL) NOPASSWD:ALL"
    shell: "/bin/bash"
final_message: |
  cloud-init has finished
  version: $version
  timestamp: $timestamp
  datasource: $datasource
  uptime: $uptime
ntp:
  enabled: false
package_update: true
package_upgrade: true
package_reboot_if_required: true
packages:
  - vim
  - rsyslog
timezone: "Europe/Amsterdam"
ssh_pwauth: false
chpasswd:
  expire: false
apt:
  primary:
    - arches: [default]
      uri: "http://deb.debian.org/debian"
  security:
    - arches: [default]
      uri: "http://deb.debian.org/debian-security"
  http_proxy: "http://updates.mydomain.local:3142/"
ca_certs:
  remove_defaults: false
  trusted:
  - |
    [redacted]
write_files:
  - path: '/root/.ssh/authorized_keys'
    owner: 'root:root'
    permissions: '0644'
    content: |
      ssh-rsa [redacted]

And the relevant parts of /var/log/cloud-init-output.log:

Cloud-init v. 25.1.4 running 'modules:final' at Mon, 13 Jul 2026 07:27:39 +0000. Up 12.34 seconds.
Get:1 file:/etc/apt/mirrors/debian.list Mirrorlist [29 B]
Get:5 file:/etc/apt/mirrors/debian-security.list Mirrorlist [38 B]
Ign:2 http://deb.debian.org/debian trixie InRelease
Ign:3 http://deb.debian.org/debian trixie-updates InRelease
Ign:4 http://deb.debian.org/debian trixie-backports InRelease
Ign:6 http://deb.debian.org/debian-security trixie-security InRelease
Ign:2 http://deb.debian.org/debian trixie InRelease
Ign:2 http://deb.debian.org/debian trixie InRelease
Ign:6 http://deb.debian.org/debian-security trixie-security InRelease
Ign:4 http://deb.debian.org/debian trixie-backports InRelease
Ign:3 http://deb.debian.org/debian trixie-updates InRelease
Err:2 http://deb.debian.org/debian trixie InRelease
  Could not resolve 'updates.mydomain.local'
Ign:3 http://deb.debian.org/debian trixie-updates InRelease
Ign:4 http://deb.debian.org/debian trixie-backports InRelease
Ign:6 http://deb.debian.org/debian-security trixie-security InRelease
Err:6 http://deb.debian.org/debian-security trixie-security InRelease
  Could not resolve 'updates.mydomain.local'
Err:4 http://deb.debian.org/debian trixie-backports InRelease
  Could not resolve 'updates.mydomain.local'
Err:3 http://deb.debian.org/debian trixie-updates InRelease
  Could not resolve 'updates.mydomain.local'
Reading package lists...
W: Failed to fetch mirror+file:/etc/apt/mirrors/debian.list/dists/trixie/InRelease  Could not resolve 'updates.mydomain.local'
W: Failed to fetch mirror+file:/etc/apt/mirrors/debian.list/dists/trixie-updates/InRelease  Could not resolve 'updates.mydomain.local'
W: Failed to fetch mirror+file:/etc/apt/mirrors/debian.list/dists/trixie-backports/InRelease  Could not resolve 'updates.mydomain.local'
W: Failed to fetch mirror+file:/etc/apt/mirrors/debian-security.list/dists/trixie-security/InRelease  Could not resolve 'updates.mydomain.local'
W: Some index files failed to download. They have been ignored, or old ones used instead.

I enabled query logging on the DNS server (unbound) and it did NOT receive any DNS queries
while running cloud-init.

Appreciate your help,

Laurens Blankers

#1141980#10
Date:
2026-07-13 17:50:53 UTC
From:
To:
I have some further information:

The bug occurs when there is a delay in network traffic being routed AND
the apt http_proxy option is used. The network delay could be due to
several issues but the most common (and easy to reproduce) is STP on the
network switch or software bridge.

In case of network delay withOUT the http_proxy option cloud-init will
retry fetching apt information logging the following message in a loop:

cloud-init[449]: W: Tried to start delayed item
http://deb.debian.org/debian trixie InRelease, but failed

Once the network is routing traffic the packages will be installed and
the installation completed successfully.

However when apt http_proxy is set the installation errors out
immediately without retrying.

The behaviour I would expect is that both with and without apt
http_proxy, cloud-init will retry until a timeout occurs (say 120
seconds). Ideally logging the "Tried to start..." line one a second
rather than as many time as possible.

Laurens