Description:

When using dhcpcd-base together with systemd-resolved, the name server settings for an interface are initially configured correctly from DHCP but may then subsequently disappear, leaving no DNS server configured.

This is because the dhcpcd hook scripts contain support for resolvconf, but the implementation of resolvconf provided by systemd-resolved is very limited and does not support the full semantics required (see below).



Steps to Reproduce:

1. Install systemd-resolved (257.5-2) along with dhcpcd-base (1:10.1.0-9) and ifupdown (0.8.44)

2. Configure an interface for DHCP in /etc/network/interfaces.
As an example I am using a usermode (slirp) interface in QEMU:

allow-hotplug enp1s0
iface enp1s0 inet dhcp

3. Reboot, or restart the interface with ifdown/ifup.

4. Observe the DNS server configuration with resolvectl, repeating a few minutes later.



Expected Behaviour:

resolvectl should show the DNS server address configured by DHCP. Name resolution should work.



Observed Behaviour:

Shortly after the interface is brought up, resolvectl does indeed show that a DNS server has been configured, e.g.:

Global
         Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 2 (enp1s0)
    Current Scopes: DNS
         Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 172.17.2.3
       DNS Servers: 172.17.2.3
     Default Route: yes


But a few minutes later, the server seems to have disappeared again and DNS is no longer working:

Global
         Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 2 (enp1s0)
    Current Scopes: none
         Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
     Default Route: no



Investigation:

Turning on the debug option for dhcpcd shows a number of hook script calls after the interface has been configured, e.g:

Apr 20 21:07:36 redacted ifup[1015]: enp1s0: executing: /usr/lib/dhcpcd/dhcpcd-run-hooks PREINIT
Apr 20 21:07:36 redacted dhcpcd[1015]: enp1s0: executing: /usr/lib/dhcpcd/dhcpcd-run-hooks PREINIT
Apr 20 21:07:36 redacted ifup[1015]: enp1s0: executing: /usr/lib/dhcpcd/dhcpcd-run-hooks NOCARRIER
Apr 20 21:07:36 redacted dhcpcd[1015]: enp1s0: executing: /usr/lib/dhcpcd/dhcpcd-run-hooks NOCARRIER
Apr 20 21:07:36 redacted ifup[1015]: enp1s0: executing: /usr/lib/dhcpcd/dhcpcd-run-hooks CARRIER
Apr 20 21:07:36 redacted dhcpcd[1015]: enp1s0: executing: /usr/lib/dhcpcd/dhcpcd-run-hooks CARRIER
Apr 20 21:07:39 redacted ifup[1015]: enp1s0: executing: /usr/lib/dhcpcd/dhcpcd-run-hooks ROUTERADVERT
Apr 20 21:07:39 redacted dhcpcd[1015]: enp1s0: executing: /usr/lib/dhcpcd/dhcpcd-run-hooks ROUTERADVERT
Apr 20 21:07:41 redacted ifup[1015]: enp1s0: executing: /usr/lib/dhcpcd/dhcpcd-run-hooks BOUND
Apr 20 21:07:41 redacted dhcpcd[1015]: enp1s0: executing: /usr/lib/dhcpcd/dhcpcd-run-hooks BOUND
Apr 20 21:12:20 redacted dhcpcd[1015]: enp1s0: executing: /usr/lib/dhcpcd/dhcpcd-run-hooks ROUTERADVERT
Apr 20 21:21:56 redacted dhcpcd[1015]: enp1s0: executing: /usr/lib/dhcpcd/dhcpcd-run-hooks ROUTERADVERT
Apr 20 21:28:29 redacted dhcpcd[1015]: enp1s0: executing: /usr/lib/dhcpcd/dhcpcd-run-hooks ROUTERADVERT

Examining the hook script /usr/lib/dhcpcd/dhcpcd-hooks/20-resolv.conf, shows that when a resolvconf binary is present, each hook call will ultimately invoke resolvconf to add or delete the DNS settings for a particular protocol on the interface. This results in a series of invocations such as the following:

resolvconf -d enp1s0.dhcp -f
resolvconf -d enp1s0.link -f
resolvconf -d enp1s0.link -f
resolvconf -d enp1s0.ra -f
resolvconf -a enp1s0.dhcp        (with a resolv.conf-formatted file piped to stdin)
resolvconf -d enp1s0.ra -f

This is where the difference in semantics becomes apparent between the versions of Debian resolvconf/openresolv and the resolvconf provided by systemd-resolved:

The "traditional" resolvconf maintains a separate state for each protocol on the interface, and generates a superposition of the information from all protocols. By contrast, the systemd version ignores the protocol suffix completely and stores a single state for all protocols. The last information supplied for any protocol typically wins out.

In the particular example above, it appears that an IPv6 router advertisement for "enp1s0.ra" has deleted the DNS configuration previously set for "enp1s0.ra". I expect that various other sequences of events may lead to a similar result.



Possible Solutions:

I attach a patch which modifies the dhcpcd hook scripts to work around the problem: We detect when the systemd version of resolvconf is being used and provide it with a resolv.conf fragment that contains the superposition of all the protocols on a given interface (retrieved from state stored under /run/dhcpcd).

I have not tested the patch extensively, although it has worked well enough in my test cases.

Note that the ifupdown and isc-dhcp-client packages already have a their own hook scripts (borrowed from Ubuntu) for dealing with systemctl-resolved. These scripts call resolvectl to set the DNS servers, and there is a shared state that is maintained by both packages so they are to some extent integrated together. Since dhcpcd has now replaced isc-dhcp-client as the default DHCP client for trixie, this integration now seems to be rather pointless. See as follows:

https://sources.debian.org/src/ifupdown/0.8.44/debian/if-up.d/resolved/
https://sources.debian.org/src/isc-dhcp/4.4.3-P1-7/debian/resolved/



Remarks:

It seems to me that the need for all these hacky hook scripts has come about because systemd's implementation of resolvconf is too limited to be of use as a drop-in replacement for resolvconf with the various "legacy" networking packages. Rather than propagating these bespoke workarounds into different packages, it may be better if they could all continue to use the "traditional" resolvconf/openresolv system to configure resolved.

I think this could be accomplished fairly cleanly and easily if the systemd-resolved package were to simply omit installing the "resolvconf" symbolic link, and drop the "Provides: resolvconf" and "Conflicts: resolvconf" dependences, allowing it to be installed alongside the "traditional" resolvconf. At the same time, resolvconf/openresolv could be equipped with a unified hook script to detect systemd-resolved and call resolvectl. Then all the hacks elsewhere could go away!

If anyone can suggest a better forum to raise that wider issue then please let me know and I will do so.

Hey Ken,

ma 21.4.2025 klo 1.42 Ken Milmore (ken.milmore@gmail.com) kirjoitti:

Thanks for this bug report and very thorough analysis.

Can you submit a report upstream and link to this bug?

https://github.com/NetworkConfiguration/dhcpcd/issues/new

Martin-Éric

Please note, however, that I don't really think this is a defect upstream:
IMHO it is really a distribution integration issue which should be solved within Debian:

- dhcpcd provides resolvconf integration, but it doesn't claim to support the crippled systemd version of resolvconf. Indeed, the same maintainer distributes openresolv for this very purpose!

- systemd provides their stand-in version of resolvconf, but it is specifically documented that it is only a "limited resolvconf compatibility mode" and "Not all operations supported by other implementations are supported natively".
See: https://manpages.debian.org/testing/systemd-resolved/resolvectl.1.en.html

- Dealing with multiple different sources of name server information requires co-operation between different packages, either by using the "proper" resolvconf, or the scripting hacks mentioned above which are shared by ifupdown and isc-dhcp-client, or some other mechanism. Any fix specific to dhcpcd will only ever be a partial solution.

Further to our upstream discussion at [1] I have raised wishlist Bug#1103785 [2] for systemd-resolved.

With trixie release coming up, I would not expect to see any enthusiasm for making these sorts of changes across multiple packages.

For now I guess you have the option of patching the hook scripts in dhcpcd or just leaving everything alone.

Best wishes,

- Ken. :-)

[1] https://github.com/NetworkConfiguration/dhcpcd/issues/500#issuecomment-2818441809
[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1103785

Hi Ken,

Thanks for raising this, I've had a hunch things are still a bit dodgy in
this area for a while now.

I'm overall not very happy with Debian's DNS configuration story and I've
been meaning to have a go at improving the situation, but so far I haven't
been able to focus on it.

If you want to have a stab at collaborating on improving the situation
consider me your condiuit. As a DD with a good grasp of Debian's "process"
and politics I can help you make the necessary changes happen.

Consider me, at least, enthused :-).

As long as potential breakage is contained to one or two packages and the
time window is reasonably small I'd consider this a "small, targeted
fix". I don't see this violating the current soft freeze policy.

We can get pre-clearance from release-team in any case, develop in
experimental and upload to unstable when ready. Business as usual.

Hard freeze on 2025-05-15 will make changes more beurocratic and slow, so
if we'd want to have a go at this it'd be best to get something together
before then.

However any DNS breakage easily qualifies for a severity:serious RC bug
which we can fix any time before the release as long as the change diff is
reasonably small and we can motivate the change to release team.

The issue I raised upstream has now been closed [1], after Roy produced a new release of openresolv [2] with support for systemd-resolved.


Unfortunately, IMHO the direction taken in openresolv doesn't help us much with the present bug, for the following reasons:

- It violates the principle of least surprise (from the POV of network interface configuration), by placing all name servers into the global pool, rather than assigning servers and their routing domains to specific network interfaces.
I am sure there are valid arguments for doing it that way. But it is completely at odds with how resolved is "normally" configured by all the other networking tools such as systemd-networkd, NetworkManager and dhclient.
By contrast, almost anyone installing systemd-resolved on Debian is going to expect per-interface configuration.

- In any case, an openresolv-based solution still wouldn't circumvent the package management issues we currently have in Debian, namely that the systemd-resolved package conflicts with anything providing resolvconf.
there's also the problem that we would have to duplicate any new openresolv behaviour into the resolvconf package to keep the two in step.


For the above reasons I've given up on the resolvconf approach, decided to eat my earlier words and reconsider the "hacky script" technique once more:

I have ported the resolved configuration script (originally borrowed from Ubuntu) from dhclient [3] to work with dhcpcd.
This turned out to be completely trivial, since the hook script environments provided by dhclient and dhcpcd are so similar.
Basically, it should just work.


See the attached patch.


Functionally speaking, this bespoke script solution has a lot to commend it:

- It configures resolved in the "expected" way, i.e. per interface, the same way as NetworkManager and the rest do.

- It integrates with ifupdown settings configured in /etc/network/interfaces:
Specifically, dns-nameservers, dns-search and dns-default-route can optionally be set to manually override what comes from DHCP.

- It is low-impact and follows the solution already in use, so we would hope for little or nothing in the way of unexpected breakage.


So what's not to like?  Well I have never been too keen on the implementation details:

- There is code duplication and shared state between the ifupdown, dhclient and (now) dhcpcd packages.
The script needs to be maintained in step between all of these packages, which implies fragility.
Strictly speaking, we could avoid this interdependency, but at the cost of removing the useful ability to override things in /etc/network/interfaces.
I think it would be preferable if the same common script could be used by all of the above packages:
Probably the central place to put it would be inside the ifupdown package. This would not necessarily require a package dependency, if things were handled correctly.

- The script modifies some internal resolved state files under /run/systemd/resolve/netif/.
I think the only reason for doing this is in case DNS gets configured before resolved has started up: I would have thought that might be better handled with systemd service dependencies or suchlike?

- There are some other minor things, such as the use of mktemp, which I would have preferred to avoid.


Notwithstanding the above wrinkles, I do think it would be worthwhile getting something along these lines into trixie:
It will allow someone who has installed a base system with ifupdown to simply "apt install systemd-resolved" and still have things work as expected.

If there is any enthusiasm for pursuing this further then I would be prepared to do some work on tweaking the solution to order.
Suggestions welcome...


[1] https://github.com/NetworkConfiguration/dhcpcd/issues/500#issuecomment-2831237831
[2] https://github.com/NetworkConfiguration/openresolv/releases/tag/v3.15.0
[3] https://salsa.debian.org/debian/isc-dhcp/-/commit/ab10fa481ee012c52de95adfc92691c882bf78ac

 ---- On Sat, 26 Apr 2025 23:27:43 +0100  Ken Milmore <ken.milmore@gmail.com> wrote --- 
 >
 > The issue I raised upstream has now been closed [1], after Roy produced a new release of openresolv [2] with support for systemd-resolved.
 >
 >
 > Unfortunately, IMHO the direction taken in openresolv doesn't help us much with the present bug, for the following reasons:
 >
 > - It violates the principle of least surprise (from the POV of network interface configuration), by placing all name servers into the global pool, rather than assigning servers and their routing domains to specific network interfaces.
 > I am sure there are valid arguments for doing it that way. But it is completely at odds with how resolved is "normally" configured by all the other networking tools such as systemd-networkd, NetworkManager and dhclient.
 > By contrast, almost anyone installing systemd-resolved on Debian is going to expect per-interface configuration.

Why are people going to expect that?
I have a counter argument - people installing stuff expect it to just work.

resolved is the *only* DNS resolver that can be configured per interface.
DNS by it's nature from configuration is global, but you can still see the interface or source where it's derived by using `resolvconf -l`

The only time it is not is when it's from a VPN and that's a whole new issue because some VPN's such as strongswan sit on top
of existing interfaces and with the current design you the have two programs (say dhclient and strongswan) wanting to update the same interface via resolvectl with competing information.
And by competing dhclient could say "here be public nameservers" and strongswan could say "here be private nameservers for this domain" and resolved currently does not support this.
This is why you need resolvconf to manage this - and by that I mean openresolv or the Debian one.
https://github.com/strongswan/strongswan/discussions/2085
systemd-resolved does not yet support Split Horizon DNS - https://github.com/systemd/systemd/issues/20485
There is a PR to resolve this via the concept of DNS Delegates from poettering - https://github.com/systemd/systemd/pull/34368
Once/if that get's merged then resolved is on feature parity with other resolvers.

Then let's talk about state. If resolved is restarted, it's state learned via resolvectl is lost. How do we get that back?
If it's baked in a config file like other DNS servers it's still there.

Lastly, let's talk about features.
openresolv offers the admin the ability to massage the resolv.conf data before it's applied to the DNS server.
You can append, prefix, remove and change pretty much anything as it comes in.
And like Debian's resolvconf you can also use it to just discard all inputs and keep /etc/resolv.conf alone.

Now, none of the above might be important to you, but it's important to others because they've opened many bugs about it.
I didn't just implement these features on a whim, they are asked for.

 >
 > - In any case, an openresolv-based solution still wouldn't circumvent the package management issues we currently have in Debian, namely that the systemd-resolved package conflicts with anything providing resolvconf.
 > there's also the problem that we would have to duplicate any new openresolv behaviour into the resolvconf package to keep the two in step.

You could split it out into it's own package, or manage it via alternatives?
https://wiki.debian.org/DebianAlternatives

 >
 >
 > For the above reasons I've given up on the resolvconf approach, decided to eat my earlier words and reconsider the "hacky script" technique once more:
 >
 > I have ported the resolved configuration script (originally borrowed from Ubuntu) from dhclient [3] to work with dhcpcd.
 > This turned out to be completely trivial, since the hook script environments provided by dhclient and dhcpcd are so similar.
 > Basically, it should just work.
 >
 >
 > See the attached patch.
 >
 >
 > Functionally speaking, this bespoke script solution has a lot to commend it:
 >
 > - It configures resolved in the "expected" way, i.e. per interface, the same way as NetworkManager and the rest do.

I find it interesting that NetworkManager does this per interface for resolved and yet just treats resolvconf as a global.

 >
 > - It integrates with ifupdown settings configured in /etc/network/interfaces:
 > Specifically, dns-nameservers, dns-search and dns-default-route can optionally be set to manually override what comes from DHCP.
 >
 > - It is low-impact and follows the solution already in use, so we would hope for little or nothing in the way of unexpected breakage.
 >
 >
 > So what's not to like?  Well I have never been too keen on the implementation details:
 >
 > - There is code duplication and shared state between the ifupdown, dhclient and (now) dhcpcd packages.
 > The script needs to be maintained in step between all of these packages, which implies fragility.
 > Strictly speaking, we could avoid this interdependency, but at the cost of removing the useful ability to override things in /etc/network/interfaces.
 > I think it would be preferable if the same common script could be used by all of the above packages:
 > Probably the central place to put it would be inside the ifupdown package. This would not necessarily require a package dependency, if things were handled correctly.

So you realise you're effectively replicating what resolvconf does here per package? This was exactly why resolvconf was born back in the day ....
You're also discarding DNS received via ND.
https://github.com/NetworkConfiguration/dhcpcd/blob/master/hooks/20-resolv.conf#L117

 >
 > - The script modifies some internal resolved state files under /run/systemd/resolve/netif/.
 > I think the only reason for doing this is in case DNS gets configured before resolved has started up: I would have thought that might be better handled with systemd service dependencies or suchlike?
 >
 > - There are some other minor things, such as the use of mktemp, which I would have preferred to avoid.
 >
 >
 > Notwithstanding the above wrinkles, I do think it would be worthwhile getting something along these lines into trixie:
 > It will allow someone who has installed a base system with ifupdown to simply "apt install systemd-resolved" and still have things work as expected.
 >
 > If there is any enthusiasm for pursuing this further then I would be prepared to do some work on tweaking the solution to order.
 > Suggestions welcome...
 >
 >
 > [1] https://github.com/NetworkConfiguration/dhcpcd/issues/500#issuecomment-2831237831
 > [2] https://github.com/NetworkConfiguration/openresolv/releases/tag/v3.15.0
 > [3] https://salsa.debian.org/debian/isc-dhcp/-/commit/ab10fa481ee012c52de95adfc92691c882bf78ac
 >

If you really want to have things pushed via resolvectl then I'm happy to revisit the resolvectl helper for openresolv as I committed here:
https://github.com/NetworkConfiguration/openresolv/commit/a21cc3bd201217e2d4e271c87e1990e03622ba99
And we could work this together to satisfy your needs.

openresolv can ship with both helpers and the admin can choose which one to use.
What do you think?

Roy

Thanks for responding Roy.

Because all of the other major networking solutions in Debian do it that way.
Try configuring an interface in NetworkManager, networkd or (lately) ifupdown with dhclient and you will see what I mean.
I'm certainly not trying to make a point about whether this is right or wrong, but it is undeniably the "usual" way that systemd-resolved gets configured by everyone else.

See above. resolved is completely optional in Debian, as are openresolv/resolvconf.
I expect they each have their own users with possibly different expectations.

Presumably this isn't getting into trixie though, and I assume NetworkManager et al will still continue to configure it the "old" way unless and until they are explicitly given support for delegates?

The "hacky patch" handles this by persisting the state under /run/network.
As I've been at pains to point out, I'm not much of a fan of the implementation, but it is already in use by ifupdown and dhclient.
Better the devil you know I suppose.

Because of the interop with ifupdown, it does have the advantage that you can override things manually from /etc/network/interfaces.
I found that particularly useful for controlling the "default route" settings in resolved, which can force it to be more pedantic about prefix matching.

At the moment, the systemd-resolved package has a hard conflict with both openresolv and resolvconf so they can never be installed together.
I have raised a feature request to get this changed, but...

If that restriction were relaxed then there are various subtleties that would need to be taken care of to avoid breaking more things:
Mostly regarding the ownership and function of /usr/sbin/resolvconf and /etc/resolv.conf with various different combinations of packages installed.
And since we have *two* different resolvconf packages, any behaviour would have to be replicated in *both* of them.

It is not uncommon in Debian desktop installs to use both ifupdown and NetworkManager at the same time, on different interfaces:
The former is most often used for wired interfaces and the latter for wireless networks.
This was another reason why I thought it was better to go per-interface with resolved, because it keeps things consistent in this sort of mixed setup.
As I've pointed out above, removing that restriction would create other problems for us to solve.
For now, doing anything that calls resolvconf in the presence of systemd-resolved is likely to remain broken. That's why I raised this bug! :-)

Cheers,

Hi Ken

So openresolv-3.16.4 will allow two approaches.
The limited per interface approach via resolvectl or the write to global and dns delegate files like any other resolver approach.
Pick your poison.
Both ways offer more resolvconf support than what systemd's resolvconf shim offers as well as anything discussed thus far and support the systemd notion of marking an entries domains as both non searchable and non routeable.

As far as trixie is concerned, both have limitations - resolvectl approach only works per interface, systemd-resolved approach relies on DNS delegate support which isn't in any systemd release yet.
Also, the resolvectl approach is slow because of the number of times we call resolvectl. A minor point, but worth noting.
strongswan - a debian package - will only work with the systemd-resolved approach as it doesn't have an interface.

I looked at this and have some issues

1) /run/systemd/resolve/netif is stated as internal private data and not part of the API
    as such any long term hack around this is a Bad Idea Which Could Break on Upgrade
2) While you can write to it and it magically appears in resolvectl after a SIGHUP, any futher action on
    this file is futile - you have to go via resolvectl. At least this is true on Fedora 41.
3) Each time you add a new hack into /run/network you need to change all the hacks

But at the end, I will say this - systemd's resolvconf shim
1) Embraces - resolvconf is a known and recognised standard of managing DNS setup from various sources (iface.prog)
2) Extends - it can configure systemd-resolved!
3) Extinguish
     - it breaks the orignal design goal of resolvconf which was to make managing different sources of DNS setup easier (rejects iface.prog, just allows iface)
     - only supports systemd-resolved

Point 3 violates the Principle Of Least Astonishment from the POV of the system administrator who has used Debian for a long time.

Roy