Dear Maintainer,

While doing unrelated storage testing for our VMware integrated product, we purposefully recreated
a storage outage by removing the iSCSI initiators from the backing array hosting the vmdk disk
images for the virtual machine.

Upon removal of uplinks to storage, the VM goes into a R/O file system state after 5-10 minutes.
When storage initiators are brought back up and the LUNs are rescanned, the VM begins to
rapidly request DHCP leases from an ISC DHCP server.  This DoS's the server in a way due
to the number of DHCPDECLINE errors, and the interface attempts to take and discard IP's in a
rapid fashion.

This only seems to appear on this distribution, and I can't replicate the behavior on Debian 9
or in a desktop environment.

One more question. When you say VNWare integrated product. AFAIK vmware
have their own networking module in the kernel? Can you reproduce this
with some other virtualisation technology like kvm, qemu?

And one more question: do depending on who does the DHCP receival in the
VM (systemd? isc-dhcp-client? [...]?): shouldn't there be some rate
limiting sanity check in the DHCP client?
*t

Could you privide a recipe on how to reproduce this? There's a lot of
very special setup below, that someone wwould need large amounts of time
to reporoduce I feel.

Is it possible to reduce the problem to something easily demonstratable?

This seems to be an important issue to me.

I think the problem here *might* be a kernel problem? Re-assign this to
kernel package?

When you say that it DoS'es the server then what does "top" say? What is
being DoS'ed? Is it the CPU?
*t

It would be truly cool, if you could provide more infos.
*t

[...]

So far as I know, the kernel only ever does DHCP if you net-boot
without an initramfs.

Ben.

Am 23.07.19 um 17:57 schrieb Ben Hutchings:

My focus was more on this issue here - aparenty:

Mark Hutchison wrote:

-> changing IPs of an interface of a *VM* can DoS the server. Which I
think is not expected, and not terribly funny. It takes a bit of not so
straightforward circumstances (as far as I can understand the bug
report), but then an attacker can DoS the server via DHCP. Which is uh,
I mean ah, um.

Information is a bit sparse here, though.

<OT>If I may shoot completely off topic for a second: Woah, many thanks
for your terrific kernel maintenance work Ben. Truly amazing :-o!!!
Thanks so may times a lot! Woah :-)!!!! Thank you! (this doesn't exclude
the rest of the kernel team - my thanks extend to you all - it's just
that I have the honor to say thanks to a participating party in this
email exchange 8v)!</OT>
*t

Hi fellas,

Apologies for the brevity in the initial bug report.  I was using the
reportbug tool directly from the console of the VM I was working on, small
resolution.  Allow me to elaborate...

We initially discovered this bug testing our storage product, we had a
Debian 10 VM running in a typical ESXi 6.7 environment with iSCSI backed
storage.  The VM ran in a VMDK file on a VMFS datastore volume.  While the
VM was running in memory, we removed the storage initiators from ESXi
purposefully to test something unrelated, to simulate a storage outage.
After a couple of minutes the OS will go into R/O mode without its disk,
and at that time dhclient will rapidly request IP's from our ISC DHCP
server.  dhclient will take the IP, consume it from the DHCP pool and then
request another.  After some period of time this depletes the DHCP pool,
several hours to days depending on the scopes size.  This could also be
replicated by deleting the hard disk from a running VM in a virtual
environment.

When I look at systemctl for the dhclient service, I can see that there's
an error, "can't create /var/lib/dhcp/dhclient.intname.leases Read Only
file system", and then the DHCPREQUEST > DHCPACK > DHCPDECLINE sequence
starts every few seconds, and occasionally the service will show "RTNETLINK
answers: File Exists."

I'm guessing from the error that dhclient has a problem with not being able
to read / write to the client leases file, declines the IP and requests
another, but secretly holds on to the IP.

The DHCP server logs will show a final DHCPDECLINE after the ACK, and mark
the address as abandoned.  The VM will still have the address leased
however.  After a period of time VMware's guest tools will show all the
consumed IP's belonging to that MAC address and virtual interface.  Network
gear ARP shows the IP's belonging to the same MAC as well.

We've consistently reproduced this bug in our lab, and performed the test
simultaneously with a Debian 9, Centos and Ubuntu 16 instance to make sure
it wasn't some kind of NetworkManager thing, or a broader Linux issue.

I see that someone reported this similar bug back in 2018 as well, I think
they may be the same thing.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=888209

Thanks, just let me know if you have any questions.

Hi fellas,

Apologies for the brevity in the initial bug report.  I was using the
reportbug tool directly from the console of the VM I was working on, small
resolution.  Allow me to elaborate...

We initially discovered this bug testing our storage product, we had a
Debian 10 VM running in a typical ESXi 6.7 environment with iSCSI backed
storage.  The VM ran in a VMDK file on a VMFS datastore volume.  While the
VM was running in memory, we removed the storage initiators from ESXi
purposefully to test something unrelated, to simulate a storage outage.
After a couple of minutes the OS will go into R/O mode without its disk,
and at that time dhclient will rapidly request IP's from our ISC DHCP
server.  dhclient will take the IP, consume it from the DHCP pool and then
request another.  After some period of time this depletes the DHCP pool,
several hours to days depending on the scopes size.  This could also be
replicated by deleting the hard disk from a running VM in a virtual
environment.

When I look at systemctl for the dhclient service, I can see that there's
an error, "can't create /var/lib/dhcp/dhclient.intname.leases Read Only
file system", and then the DHCPREQUEST > DHCPACK > DHCPDECLINE sequence
starts every few seconds, and occasionally the service will show "RTNETLINK
answers: File Exists."

I'm guessing from the error that dhclient has a problem with not being able
to read / write to the client leases file, declines the IP and requests
another, but secretly holds on to the IP.

The DHCP server logs will show a final DHCPDECLINE after the ACK, and mark
the address as abandoned.  The VM will still have the address leased
however.  After a period of time VMware's guest tools will show all the
consumed IP's belonging to that MAC address and virtual interface.  Network
gear ARP shows the IP's belonging to the same MAC as well.

We've consistently reproduced this bug in our lab, and performed the test
simultaneously with a Debian 9, Centos and Ubuntu 16 instance to make sure
it wasn't some kind of NetworkManager thing, or a broader Linux issue.

I see that someone reported this similar bug back in 2018 as well, I think
they may be the same thing.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=888209

Thanks, just let me know if you have any questions.

As soon as the filesystem its lease file resides on becomes unreachable
or read-only, it throws a fit and just hammers away at the DHCP
infrastructure.

In our case every client has a fixed DHCP reservation and only ever gets
OFFERed the same IP, which he then declines, but when you have several
hundred clients flooding DHCP reequests at the same time, the load on
the infrastructure, including switches with DHCP Snooping active, is
immense.

I also think that #888209 is the same issue.

Coincidentally it also happened in out VMware cluster when an
iSCSI-backed LUN when down but you should be easily able to reproduce
this with a simple local KVM setup.

Grüße,
Sven.

So my interpretation of your initial bug report, that the VM would DoS the
host on which it was running via fast changing of IP addresses on its
interface was completely off the track?

So what you wanted in fact wanted to say by "DoS'ing the server" was that
the VM sends huge amounts of DHCP requests to the DHCP server (possibly
also in addition depleting IP addresses from the DHCP server's IP address
pool) and *that* amounts to a DoS? Is my interpretation correct?

If that's the case, then I'm reassinging this bug report to
isc-dhcp-client and merging it with the mentioned bug report #888209.

*t

While I am not the original reporter of #932769, from the observation I
made in my infrastructure: Yes, your interpretation is correct.

The isc-dhcp-client goes bonkers and just floods DISCOVERs into the
network, directly DECLINEing any IP it gets OFFERed from the DHCP
server. (I've seen some 600,000 Requests over the span of 12 hours from
only one system. Multiply this by several systems and you get the
possible impact.)

Grüße,
Sven.

Hi Tomas,

Thanks for recategorizing the bug and applying the version.  I was getting
caught up and see you already did this.

Just wanted to add a few points from additional testing I've been doing.
filling the volume up until no writes can be performed, but requesting a
new lease behaves as normal.  I think swap has something to do with this
since the file system is still mounted R/W.  I've also tried deleting the
leases file and chmoding that directory to RO.  It appears that storage
containing the OS has to truly disconnect for some time and cause the
volume to be marked as RO.
platform.  I'll test on virtualbox and KVM and let you know if anything is
different there.

Thanks, let me know if I can answer any questions.
-Mark

First I can confirm this bug. Some time ago I had two servers which
caused a huge amount of dhcp requests to log the DHCP server (managed
by a different departement). They told me one server did 30 requests
per seconds.


I've looked at the CentOS 8 sources and found this patch:

The patch is in the directory SOURCES called
dhcp-dhclient-decline-backoff.patch. Since I'm not good in C
programming it would be good I someone else can have a look if Debian
should include this patch. I'll attach the patch from Centos.

looking at the code, upstream seems to have seen this problem as well
and changed the client to wait for 10 seconds after a DECLINE before
doing a DISCOVER again.

This was ISC-Bug 45457:
https://bugs.isc.org/Public/Bug/Display.html?id=45457

This bug has been fixed in ISC DHCP 4.3.6.

The commit implementing this is
https://gitlab.isc.org/isc-projects/dhcp/commit/3e88222f1c2f7a365b9fde018bb4bf86520b51d6

So for Debian this problem only exists in oldoldstable (Jessie) and
oldstable (Stretch).

In my opinion this change should be pulled into Stretch and possibly Jessie.

Grüße,
Sven.

I misread, it was fixed for 4.4.0. The commit is the correct one though.

Grüße,
Sven.

    > This bug has been fixed in ISC DHCP 4.3.6.
I had the problem with Debian stretch. Since I'm moving to buster soon,
this is not that important for me. But it may be nice to have this fixed
for stretch.

    > In my opinion this change should be pulled into Stretch and possibly Jessie.
I think stretch is sufficient.

Hi,

We experience this issue too, in Bullseye and Bookworm (updated from
Bullseye).

Our environment is similar to that described by Mark (the initial
reporter), i.e., a VMware ESXi cluster with VMs stored on a network
storage. In our case, if the network storage becomes full, VMs get disk
IO errors and mark file systems as read-only.

After that, the DHCP client starts requesting and declining IP addresses
in a rapid manner. Leased addresses stay configured on the VMs' network
interfaces, which leads to DHCP pool exhaustion.

What happens in details:

- The DHCP client requests and receives an IP address.
- It runs the script /sbin/dhclient-script.
- The script configures a network interface.
- Next the script updates the /etc/resolv.conf file.
  For this, it calls the wait_for_rw function to check if it can write
  to /etc/resolv.conf.<shell PID>.
- If a file system is read only this check doesn't work as expected.
  The script fails and exits with a non-zero code once it hits the
  following line in wait_for_rw():

    while ! { : >> "$file"; } 2>/dev/null;

- So the script exits, but the network interface still has leased IP
  address.
- When DHCP client gets a non-zero code from the script, it sends
  DHCPDECLINE to a DHCP server.
- The DHCP server abandons the IP address. It might try to reuse it
  later by performing a ping check but the misbehaving DHCP client will
  respond to it.

The script is executed by Dash (being /bin/sh) and its behavior complies
with POSIX:
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_08_01,
"Redirection error with special built-in utilities" - "shall exit" for
non-interactive shells.

The error doesn't occur if the script executed by Bash. Currently we
test a simple Bash wrapper script as a workaround:

  #!/bin/bash
  source /sbin/dhclient-script

The problem can be reproduced by issuing "echo 1 >
/sys/block/sda/device/delete" on a test machine to remove the disk (sda
here) from the kernel.