- Package:
- cloud-init
- Source:
- cloud-init
- Submitter:
- Zar VPN
- Date:
- 2025-05-04 06:21:02 UTC
- Severity:
- normal
- Tags:
Subject: cloud-init fails to handle IPv6-only environments in Debian 12 generic-cloud Dear Debian Development Team, I am writing to report an issue with the cloud-init package in the current version of Debian 12 (Bookworm). In an IPv6-only environment, cloud-init fails to properly detect and interact with the metadata service, which prevents the successful initialization of cloud instances, including login access via SSH or other remote methods. ### Issue Details: - **System:** Debian 12 (Bookworm) cloud image (2025-04-28 20:40) - **Problem:** Cloud-init fails to configure instances properly in an IPv6-only environment. - **Cause:** It appears the version of cloud-init included in Debian 12 (Bookworm) is outdated and does not fully support IPv6-only networking configurations. ### Steps to Reproduce: 1. Launch a cloud instance using the official Debian 12 cloud image (genericcloud) in an environment that has IPv6-only networking. 2. Ensure that the metadata service is accessible via IPv6 (e.g., `http:// [fe80::a9fe:a9fe%<interface>]/openstack`). 3. Observe that cloud-init attempts to fetch metadata but fails because it cannot handle the IPv6-only configuration properly. 4. Testing `curl -6` works correctly to retrieve metadata using the appropriate IPv6 address, confirming that the server and networking are configured correctly. 5. Cloud-init continues to default to IPv4 or fails to detect the IPv6 configuration. ### Observed Behavior: The root cause seems to be that the version of cloud-init shipped with Debian 12 does not properly handle IPv6-only networks, particularly when it comes to metadata fetching from an IPv6 address. Even though the metadata service is reachable via IPv6, cloud-init fails to detect and use this configuration due to its outdated handling of IPv6 routing and network settings. ### Proposed Solution: Upon investigation, it appears that updating the cloud-init package to the latest version would resolve this issue. The latest versions of cloud-init include improved support for IPv6-only environments and should handle the metadata service correctly when using IPv6 addressing. I kindly request that the cloud-init package be updated to the latest stable version that includes better IPv6-only support. This will ensure that Debian 12 instances function correctly in modern cloud environments where IPv6-only networking is the default. ### Additional Information: - The issue has been reproduced on multiple Debian 12 cloud images (genericcloud). - The metadata service is working correctly when accessed via `curl -6` on the host. - The problem is specific to cloud-init's interaction with IPv6-only environments and is not related to network misconfiguration or the metadata service itself. Thank you for your attention to this issue. Please feel free to contact me if you need additional information or further clarification. I look forward to your response and the resolution of this issue. Best regards
This is not impossible, but unlikely to happen at this point in the Debian 12 lifecycle. However, Debian 13 will be released later this year. If you're able to test the current prerelease images (see https://cloud.debian.org/images/cloud/trixie/daily/) I'd love to hear how it works. noah
This is not impossible, but unlikely to happen at this point in the Debian 12 lifecycle. However, Debian 13 will be released later this year. If you're able to test the current prerelease images (see https://cloud.debian.org/images/cloud/trixie/daily/) I'd love to hear how it works. noah
Thank you for your response. As requested, I tested the latest daily Trixie (Debian 13) cloud image from https://cloud.debian.org/images/cloud/trixie/daily/. Unfortunately, the same issue persists — cloud-init still fails to properly fetch metadata in an IPv6-only environment, even though the metadata service is reachable via IPv6 using `curl -6`. This suggests that the root cause is not resolved in the current `cloud-init` version shipped with Debian 13 daily images. It appears that cloud-init does not fully handle link-local IPv6 addresses with the interface scope (e.g., `%eth0`), or fails to prioritize IPv6 when no IPv4 is present. This is especially problematic in modern cloud deployments where IPv6-only networking is common. Would you like a new bug report for Debian 13 (Trixie), or should we continue tracking the issue under this report? Please let me know how I can assist further in debugging or reproducing the issue. Best regards
root@localhost:/home/debian# cat /etc/netplan/50-cloud-init.yaml
network:
version: 2
ethernets:
enp3s0:
match:
macaddress: "fa:16:3e:b3:91:7c"
dhcp4: true
dhcp6: true
set-name: "enp3s0"
root@localhost:/home/debian# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group
default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state
UP group default qlen 1000
link/ether fa:16:3e:b3:91:7c brd ff:ff:ff:ff:ff:ff
altname enxfa163eb3917c
inet6 2a03:90c0:92:1::51d/128 scope global dynamic noprefixroute
valid_lft 85973sec preferred_lft 85973sec
inet6 fe80::f816:3eff:feb3:917c/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
root@localhost:/home/debian# ifconfig
enp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::f816:3eff:feb3:917c prefixlen 64 scopeid 0x20<link>
inet6 2a03:90c0:92:1::51d prefixlen 128 scopeid 0x0<global>
ether fa:16:3e:b3:91:7c txqueuelen 1000 (Ethernet)
RX packets 220045 bytes 49438029 (47.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 5976 bytes 592868 (578.9 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 80 bytes 6480 (6.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 80 bytes 6480 (6.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
root@localhost:/home/debian# ip r
root@localhost:/home/debian# ip -6 r
2a03:90c0:92::/48 dev enp3s0 proto ra metric 100 expires 2591871sec pref
medium
2a03:90c0:92::/48 dev enp3s0 proto kernel metric 256 expires 2591433sec
pref medium
fe80::/64 dev enp3s0 proto kernel metric 256 pref medium
default nhid 40154665 via fe80::21c:73ff:fe00:75ff dev enp3s0 proto ra
metric 100 expires 1671sec pref medium
root@localhost:/home/debian# ip -6 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
inet6 2a03:90c0:92:1::51d/128 scope global dynamic noprefixroute
valid_lft 85860sec preferred_lft 85860sec
inet6 fe80::f816:3eff:feb3:917c/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
root@localhost:/home/debian# rdisc6 -m enp3s0
Soliciting ff02::2 (ff02::2) on enp3s0...
Hop limit : 64 ( 0x40)
Stateful address conf. : Yes
Stateful other conf. : No
Mobile home agent : No
Router preference : medium
Neighbor discovery proxy : No
Router lifetime : 1800 (0x00000708) seconds
Reachable time : unspecified (0x00000000)
Retransmit time : 1000 (0x000003e8) milliseconds
Source link-layer address: 00:1C:73:00:75:FF
MTU : 1500 bytes (valid)
Prefix : 2a03:90c0:92::/48
On-link : Yes
Autonomous address conf.: Yes
Valid time : 2592000 (0x00278d00) seconds
Pref. time : 604800 (0x00093a80) seconds
from fe80::21c:73ff:fe00:75ff
root@localhost:/home/debian# curl -6 http://
[fe80::a9fe:a9fe%25enp3s0]/openstack
2012-08-10
2013-04-04
2013-10-17
2015-10-15
2016-06-30
2016-10-06
2017-02-22
2018-08-27
2020-10-14
latestroot@localhost:/home/debian# curl -6
http://[fe80::a9fe:a9fe%25enp3s0]/openstack
-v
* Trying [fe80::a9fe:a9fe]:80...
* Connected to fe80::a9fe:a9fe (fe80::a9fe:a9fe) port 80
* using HTTP/1.x
* Request completely sent off
< HTTP/1.1 200 OK
< Content-Type: text/plain; charset=UTF-8
< Content-Length: 105
< Date: Wed, 30 Apr 2025 09:55:47 GMT
<
2012-08-10
2013-04-04
2013-10-17
2015-10-15
2016-06-30
2016-10-06
2017-02-22
2018-08-27
2020-10-14
* Connection #0 to host fe80::a9fe:a9fe left intact
latestroot@localhost:/home/debian# cloud-init --version
/usr/bin/cloud-init 25.1.1
root@localhost:/home/debian# ping google.com
PING google.com (2a00:1450:4001:81c::200e) 56 data bytes
64 bytes from lcfraa-aa-in-x0e.1e100.net (2a00:1450:4001:81c::200e):
icmp_seq=1 ttl=117 time=0.949 ms
64 bytes from lcfraa-aa-in-x0e.1e100.net (2a00:1450:4001:81c::200e):
icmp_seq=2 ttl=117 time=0.959 ms
64 bytes from lcfraa-aa-in-x0e.1e100.net (2a00:1450:4001:81c::200e):
icmp_seq=3 ttl=117 time=0.903 ms
64 bytes from lcfraa-aa-in-x0e.1e100.net (2a00:1450:4001:81c::200e):
icmp_seq=4 ttl=117 time=0.988 ms
^C
--- google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3023ms
rtt min/avg/max/mdev = 0.903/0.949/0.988/0.030 ms
root@localhost:/home/debian#
root@localhost:/home/debian# rm -rf /var/log/cloud-init*
root@localhost:/home/debian# cloud-init clean
root@localhost:/home/debian# cloud-init init
Cloud-init v. 25.1.1 running 'init' at Wed, 30 Apr 2025 09:58:59 +0000. Up
930.17 seconds.
ci-info: +++++++++++++++++++++++++++++++++++++Net device
info+++++++++++++++++++++++++++++++++++++
ci-info:
+--------+------+------------------------------+-----------+--------+-------------------+
ci-info: | Device | Up | Address | Mask | Scope
| Hw-Address |
ci-info:
+--------+------+------------------------------+-----------+--------+-------------------+
ci-info: | enp3s0 | True | 2a03:90c0:92:1::51d/128 | . |
global | fa:16:3e:b3:91:7c |
ci-info: | enp3s0 | True | fe80::f816:3eff:feb3:917c/64 | . | link
| fa:16:3e:b3:91:7c |
ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host
| . |
ci-info: | lo | True | ::1/128 | . | host
| . |
ci-info:
+--------+------+------------------------------+-----------+--------+-------------------+
ci-info: ++++++++++++++++++++++++++++++Route IPv6
info+++++++++++++++++++++++++++++++
ci-info:
+-------+-------------------+--------------------------+-----------+-------+
ci-info: | Route | Destination | Gateway | Interface
| Flags |
ci-info:
+-------+-------------------+--------------------------+-----------+-------+
ci-info: | 0 | 2a03:90c0:92::/48 | :: | enp3s0
| Ue |
ci-info: | 1 | 2a03:90c0:92::/48 | :: | enp3s0
| Ue |
ci-info: | 2 | fe80::/64 | :: | enp3s0
| U |
ci-info: | 3 | ::/0 | fe80::21c:73ff:fe00:75ff | enp3s0
| UGe |
ci-info: | 5 | local | :: | enp3s0
| U |
ci-info: | 6 | local | :: | enp3s0
| U |
ci-info: | 7 | multicast | :: | enp3s0
| U |
ci-info:
+-------+-------------------+--------------------------+-----------+-------+
2025-04-30 09:59:03,895 - log_util.py[WARNING]: No active metadata service
found
Generating public/private rsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_rsa_key
Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub
The key fingerprint is:
SHA256:rwM0qj33ZfxQQdRZLflgDcuvQkSj+YCyY8KBRuuJm+w root@localhost
The key's randomart image is:
+---[RSA 3072]----+
| . .=..*o|
| . o . = oB.o|
| + . . . + o.o+ |
| + o .oo + . ..|
|. o oo+.S + .|
|.o .o.. o o . |
|o. o . * . . |
|. . o . .+ o . |
| E o .o. . |
+----[SHA256]-----+
Generating public/private ecdsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_ecdsa_key
Your public key has been saved in /etc/ssh/ssh_host_ecdsa_key.pub
The key fingerprint is:
SHA256:AdKwWkG8rXJvLZ8G5yy8o14jG0P+JQhds/ExqbDbd5U root@localhost
The key's randomart image is:
+---[ECDSA 256]---+
| o=o. |
| .+.. . |
| +o+ = |
| +.+.* + . |
| o +.o S E |
| .+o+. . . |
| oBo=*o . |
| BB==o |
| .+ooBo |
+----[SHA256]-----+
Generating public/private ed25519 key pair.
Your identification has been saved in /etc/ssh/ssh_host_ed25519_key
Your public key has been saved in /etc/ssh/ssh_host_ed25519_key.pub
The key fingerprint is:
SHA256:Lap1Ym0qy/Q6ZEYSfuwaU1/8TTpSABMzrfA0ep1gL8g root@localhost
The key's randomart image is:
+--[ED25519 256]--+
| *+ |
| . . =+o |
| . + B B o |
| o E = *.. . |
| * o oSo.+ |
| o = .o..+ . |
| B. = +. . |
| .oo= = |
| ==o |
+----[SHA256]-----+
root@localhost:/home/debian#
Since it sounds like you're experiencing this in an OpenStack cloud, one workaround would be to boot your server instance with configdrive enabled; cloud-init knows how to get its metadata that way as well.
Correct, however, this is still an issue in the Debian image that deserves a fix. Thomas Goirand (zigo)
Dear Jeremy, Thank you for your response and the suggestion to enable configdrive. OpenStack-OVS Unfortunately, this approach is not applicable in our case. We are using the official `debian-12-genericcloud-amd64` image, which is specifically designed for cloud environments and intentionally excludes certain hardware-related drivers. Based on available information, the `genericcloud` images lack the necessary drivers for accessing CD-ROM devices. This limitation has been confirmed by Debian developers. Notably, you yourself mentioned in a previous post to the debian-cloud mailing list that these images are targeted for environments such as AWS and Azure, and as a result, CD-ROM drivers were deliberately excluded: not require CD-ROM access, and therefore CD-ROM drivers are not included.” Reference: https://lists.debian.org/debian-cloud/2024/11/msg00010.html More importantly, our concern here is not merely finding a workaround, but addressing a **functional bug** in cloud-init when running in an IPv6-only environment. While alternative methods might exist under specific conditions, this bug prevents the system from initializing properly when metadata is only available via IPv6. This effectively renders the image unusable in any IPv6-only deployment scenario, making the issue **extremely high priority**, as it blocks access to core cloud functionality. Thank you again for your time and support. Best regards
Yeah, and OpenStack is only listed with exceptions. The image contains support from cdrom, as used a different way on Azure, but it lacks support for AHCI (maybe we should finally change that, as this pops up again and again). But you could try to set "hw_cdrom_bus=virtio" on the image. Bastian
----- Forwarded message from Bastian Blank <waldi@debian.org> ----- Date: Thu, 1 May 2025 10:49:02 +0200 Yeah, and OpenStack is only listed with exceptions. The image contains support from cdrom, as used a different way on Azure, but it lacks support for AHCI (maybe we should finally change that, as this pops up again and again). But you could try to set "hw_cdrom_bus=virtio" on the image. Bastian----- End forwarded message -----
I just had to enable some physical storage drivers at $day job. lsilogic, mptsas3, and ahci cover the defaults from old vmware, proxmox, and eve os. Unpacked size was only ~2M. Ross
Indeed, this is a common mistake and we'd be better with AHCI CD support. Performance on the confg drive isn't important anyways. Thomas Goirand (zigo)
Indeed, this is a common mistake and we'd be better with AHCI CD support. Performance on the confg drive isn't important anyways. Thomas Goirand (zigo)
Dear all, Thank you to everyone contributing to this discussion. I would like to gently emphasize that the core issue reported in this bug is the failure of cloud-init to fetch metadata over an IPv6-only network, specifically when using the debian-12-genericcloud-amd64 image. This results in a complete failure to initialize the instance in such environments. While I appreciate the suggestions involving configdrive, However, this is a side topic. The bug here is not about configdrive support, but rather about the inability of cloud-init to function properly over an IPv6-only metadata service — something that is expected to work. This is a critical issue, as it prevents users from booting and configuring instances in modern IPv6-only cloud environments using the official Debian cloud image. I sincerely hope we can refocus on diagnosing and resolving the actual problem. I remain available to provide any technical detail as needed. Warm regards.
Control: tags -1 + upstream
Control: forwarded -1 https://github.com/canonical/cloud-init/issues/6205
It seems that it's either cloud-init itself or python's HTTP client
(urllib3 and/or requests).
cloudinit/sources/DataSourceOpenStack.py defines a function
wait_for_metadata_service(). This contains the default list of IMDS endpoints:
DEF_MD_URLS = [
"http://[fe80::a9fe:a9fe%25{iface}]".format(
iface=self.distro.fallback_interface
),
"http://169.254.169.254",
]
urls = self.ds_cfg.get("metadata_urls", DEF_MD_URLS)
It constructs a list of URLs to probe when looking for a functioning
IMDS endpoint by appending the "openstack" path to the default list of
endpoints, as well as any passed in the configuration:
for url in urls:
md_url = url_helper.combine_url(url, "openstack")
md_urls.append(md_url)
It then probes those endpoints:
avail_url, _response = url_helper.wait_for_url(
urls=md_urls,
max_wait=url_params.max_wait_seconds,
timeout=url_params.timeout_seconds,
connect_synchronously=False,
)
However, it doesn't actually seem to be able to successfully probe a
link-local endpoint at all. We can test this ourselves by constructing
a simplified test case:
noahm@foo:~$ cat /tmp/t.py
#!/usr/bin/python3
from cloudinit import url_helper
url="http://[fe80::a9fe:a9fe%enp0s1]"
md_url = url_helper.combine_url(url, "openstack")
md_urls=[md_url]
print(url_helper.wait_for_url(md_urls, max_wait=5, timeout=1))
noahm@foo:~$ python3 /tmp/t.py
(False, None)
Both the server logs and tcpdump show no request is ever issued to the
given URL.
But if we change that to use a globally scoped address, it works:
noahm@foo:~$ cat /tmp/t.py
#!/usr/bin/python3
from cloudinit import url_helper
# url="http://[fe80::a9fe:a9fe%enp0s1]"
url="http://[fd00:80db:0:5:34e5:8aff:fec5:b9bf]"
md_url = url_helper.combine_url(url, "openstack")
md_urls=[md_url]
print(url_helper.wait_for_url(md_urls, max_wait=5, timeout=1))
noahm@foo:~$ python3 /tmp/t.py
('http://[fd00:80db:0:5:34e5:8aff:fec5:b9bf]/openstack', b'<!doctype html>\n<html>\n<head>\n <title>untitled</title>\n</head>\n<body>\n</body>\n</html>\n')
And to be sure, the server does reply to queries on link-local
addresses:
noahm@foo:~$ curl -v 'http://[fe80::a9fe:a9fe%enp0s1]/openstack'
* Trying [fe80::a9fe:a9fe]:80...
* Connected to fe80::a9fe:a9fe (fe80::a9fe:a9fe) port 80
* using HTTP/1.x
< HTTP/1.1 301 Moved Permanently
< Server: nginx/1.22.1
< Date: Sun, 04 May 2025 02:43:32 GMT
< Content-Type: text/html
< Content-Length: 169
< Location: http://[fe80::a9fe:a9fe]/openstack/
< Connection: keep-alive
<
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.22.1</center>
</body>
</html>
* Connection #0 to host fe80::a9fe:a9fe left intact
We can also see evidence suggesting that something is wrong in
cloud-init from the logs you provided:
2025-04-30 09:59:03,739 - url_helper.py[DEBUG]: [0/1] open 'http://[fe80::a9fe:a9fe%25enp3s0]/openstack' with {'url': 'http://[fe80::a9fe:a9fe%25enp3s0]/openstack', 'stream': False, 'allow_redirects': True, 'method': 'GET', 'timeout': 10.0, 'headers': {'User-Agent': 'Cloud-Init/25.1.1'}} configuration
2025-04-30 09:59:03,893 - url_helper.py[DEBUG]: [0/1] open 'http://169.254.169.254/openstack' with {'url': 'http://169.254.169.254/openstack', 'stream': False, 'allow_redirects': True, 'method': 'GET', 'timeout': 10.0, 'headers': {'User-Agent': 'Cloud-Init/25.1.1'}} configuration
2025-04-30 09:59:03,895 - url_helper.py[DEBUG]: Exception(s) [UrlError('HTTPConnectionPool(host=\'fe80::a9fe:a9fe%25enp3s0\', port=80): Max retries exceeded with url: /openstack (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x7fb44b82fe00>: Failed to resolve \'fe80::a9fe:a9fe%25enp3s0\' ([Errno -2] Name or service not known)"))'), UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /openstack (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb44b6d9810>: Failed to establish a new connection: [Errno 101] Network is unreachable'))")] during request to http://169.254.169.254/openstack, raising last exception
2025-04-30 09:59:03,895 - url_helper.py[DEBUG]: Calling 'http://169.254.169.254/openstack' failed [0/-1s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /openstack (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb44b6d9810>: Failed to establish a new connection: [Errno 101] Network is unreachable'))]
2025-04-30 09:59:03,895 - DataSourceOpenStack.py[DEBUG]: Giving up on OpenStack md from ['http://[fe80::a9fe:a9fe%25enp3s0]/openstack', 'http://169.254.169.254/openstack'] after 0 seconds
2025-04-30 09:59:03,895 - log_util.py[WARNING]: No active metadata service found
2025-04-30 09:59:03,895 - log_util.py[DEBUG]: No active metadata service found
Note in particular this:
Exception(s) [UrlError('HTTPConnectionPool(host=\'fe80::a9fe:a9fe%25enp3s0\', port=80): Max retries exceeded with url: /openstack (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x7fb44b82fe00>: Failed to resolve \'fe80::a9fe:a9fe%25enp3s0\' ([Errno -2] Name or service not known)"))')
There shouldn't be any name resolution involved here at all. My guess
is that something is not recognizing the scoped link-local address as an
IP address, and is treating it as a hostname that needs to be resolved
in DNS. Which is obviously going to fail. I haven't looked deeply
enough to determine whether this is cloud-init or a lower-level http
client.
noah
"requests" quotes the whole url, so undoes the fixup for "%25" to "%". urllib3 then does not de-quote the hostname, so "%25" is given to "getaddrinfo". Bastian
A bit further: requests.adapters._urllib3_request_context uses urllib.parse.urlparse, which then returns the quoted form, but without []. urllib3.connectionpool._normalize_host will only dequote if it finds the [] before removing them. Bastian