I believe I spotted a race condition in virtio-net or qemu/kvm (but
only when virtio-net is involved).
To replicate, one needs a virtualization environment similar to
Host:
- debian 12 x86_64
- caching name server listening on 127.0.0.1
Guest:
- linux/musl or linux/glibc or freebsd or openbsd
- kvm acceleration
- virtio netdev, configured in (default) user-mode
- /etc/resolv.conf:
nameserver 10.0.2.2 i.e. the caching dns in the host
nameserver 192.168.1.123 non existent
and run the attached program in the guest.
The program opens a UDP socket, sends out a bunch of (dns) requests,
poll()s on the socket, and then receives the responses.
If a delay is inserted between the sendto() calls, the (unique) response
from the host is received correctly:
$ ./a.out 10.0.2.2 >/dev/null # to warm up the host cache
$ ./a.out 10.0.2.2 delay 192.168.1.123
poll: 1 1 1
recvfrom() 45
<response packet>
recvfrom() -1
If the sento()s are performed in short order, the response packet
gets lost:
$ ./a.out 10.0.2.2 192.168.1.123
poll: 0 1 0
recvfrom() -1
recvfrom() -1
A tcpdump capture on the host side shows no difference between the two cases.
Tcpdump on the guest side is another story: in the good case, it looks like
this
7:32:44.332 IP 10.0.2.15.43276 > 10.0.2.2.53: 33452+ A? example.com. (29)
7:32:44.333 IP 10.0.2.2.53 > 10.0.2.15.43276: 33452 1/0/0 A 93.184.216.34 (45)
7:32:44.349 IP 10.0.2.15.43276 > 192.168.1.123.53: 33452+ A? example.com. (29)
while in the bad case it looks like this
7:32:55.358 IP 10.0.2.15.46537 > 10.0.2.2.53: 33452+ A? example.com. (29)
7:32:55.358 IP 10.0.2.15.46537 > 192.168.1.123.53: 33452+ A? example.com. (29)
7:32:55.358 IP *127.0.0.1*.53 > 10.0.2.15.46537: 33452 1/0/0 A 93.184.216.34 (45)
where the response packet has wrong src ip.
Looks like a failure of the NAT layer, but it does not happen when
the guest uses another emulated network driver: don't know whether it's
because the relevant code is in virtio-net or because other drivers add
overhead that masks the issue.
There's nothing special in port 53: I was just investigating
a weird failure in name resolution in a MUSL based guest
(https://www.openwall.com/lists/musl/2024/02/17/3) and wrote the program
to mimic MUSL resolver's behaviour.
But it succeeds/fails consistently with a different port, and in all
guests I tried (as long as the emulated network device is virtio-net).
To see the issue, it's important that the response to the first request
is so fast that it's simultaneous with the second request.
Best regards,
g.b.