Previously discussed on the mailing list, which led to a request for
concrete Policy language.

Over the years, "Essential" has made it difficult to reduce installation
size, to reduce chroot/container size, or to coordinate various
transitions. Removing something from the Essential set requires tracking
down every package using it, adding a dependency, carefully managing a
transition across Debian releases, and risking breakage of third-party
packages outside Debian.

Add language to Policy that still allows existing Essential packages to
refactor or transition to different packages (coordinated via
debian-devel) but does not allow new Essential packages.

Also add language that allows packages to declare dependencies on
Essential packages as part of transitions (such dependencies would
previously violate Policy, technically), and soften language that
connects "base system" with "essential".

This change does not propose eliminating the concept of Essential, nor
does it propose that any specific package become non-Essential.

Patch attached; also available at
https://salsa.debian.org/josh-guest/policy/ on the branch
no-new-essential.

- Josh Triplett

I do not think this proposal make sense _as a Debian policy change_.
What I mean is that if the release team decide that some new packages
need to be marked Essential: yes for some technical reason, then either
policy will be ignored and this change will be reverted. Or maybe the
bug will be serious but not RC.

Remember as Manoj used to say, policy is no a stick to beat people with.

Cheers,
Bill.

Hello,

I'm inclined to agree with Bill here.

In particular, it is not clear to me we have a project consensus that
the requirement to seek consensus on debian-devel is not sufficient.

I concur with the comments raised so far.

I think it would be great to do a better job of outlining the problems
with essential packages in debian-policy.
I don't understand why we would tie our hands though.
A consensus of debian-devel seems adequate to consider those issues and
evaluate them.
If after making that consideration we decide to create a new essential
package, we're going to do that policy not withstanding.

I would support a statement in policy that as of the time of writing we
do not anticipate ever creating a new essential package.  That would
help people considering proposing making an essential package know they
are probably looking at things the wrong way.

...

But is it an actual problem ?
Do we see packages marked Essential: yes by mistake ?

Each time we make policy longer we dilute the content.

Cheers,

    Bill> But is it an actual problem ?  Do we see packages marked
    Bill> Essential: yes by mistake ?

I think Josh's analysis brought up some important points for me that I
did not consider before and that need to be considered making decisions
in the future.
I think capturing that analysis in a pointer from policy or in policy is
important to me.

    Bill> Each time we make policy longer we dilute the content.


Obviously, in the limit this is true.
I think that capturing rationale for things that have good rationale and
that could affect the project if not properly considered is worth doing
even though it makes policy longer.

What is the point of Essential? To omit declaring dependencies on the
false assumption that some packages are always required by all systems;
the concept is essentially ill. Thus, if you are not pursuing the
elimination of Essential, your effort is virtually useless. Do you want
to remove Essential?

What is the point of Essential? To omit declaring dependencies on the
false assumption that some packages are always required by all systems;
the concept is essentially ill. Thus, if you are not pursuing the
elimination of Essential, your effort is virtually useless. Do you want
to remove Essential?

El dl 21 de 09 de 2020 a les 17:15 +0200, Javier Serrano Polo va
escriure:

Since it looks like you do not try to eliminate Essential, I will close
this report.

El dl 21 de 09 de 2020 a les 17:15 +0200, Javier Serrano Polo va
escriure:

Since it looks like you do not try to eliminate Essential, I will close
this report.

I want to avoid letting the problem get any worse. Eliminating the
concept entirely would have substantial backwards-compatibility issues,
and would invite substantially more controversy. Right now, I'd just
like to propose not adding any *new* Essential packages.

I want to avoid letting the problem get any worse. Eliminating the
concept entirely would have substantial backwards-compatibility issues,
and would invite substantially more controversy. Right now, I'd just
like to propose not adding any *new* Essential packages.

I want to avoid letting the problem get any worse. Eliminating the
concept entirely would have substantial backwards-compatibility issues,
and would invite substantially more controversy. Right now, I'd just
like to propose not adding any *new* Essential packages.

El dt 29 de 09 de 2020 a les 15:08 -0700, Josh Triplett va escriure:

So Essential packages are a problem. Do you want to remove Essential in
the long-term? If this goal is not clear, there is little point in
changing policy. New Essential packages will exist if there is a good-
enough reason.

El dt 29 de 09 de 2020 a les 15:08 -0700, Josh Triplett va escriure:

So Essential packages are a problem. Do you want to remove Essential in
the long-term? If this goal is not clear, there is little point in
changing policy. New Essential packages will exist if there is a good-
enough reason.

Hi,

Josh Triplett wrote:

Interesting.  On the other side if we were to eliminate Essential
would be bloat in the Packages file from e.g. ~every package needing
Pre-Depends on a shell.

[...]

I think I'd be more supportive of this change if it did.  Freezing the
current essential set in time feels oddly unpragmatic.  If we had a
plan, even one that would take a while, to shrink the essential set,
then it would be more likely to feel worth the cognitive load.

I think there is still a problem to solve here --- e.g. maybe there is
some definition of essential that we may want to move to that would
include things like base-files but wouldn't include things like dpkg
(to take an extreme example) but I don't think we've found it yet.

So even though I'm a fan of the intent here, I agree with the
consensus that we should close this until we have a more specific
proposal.

Thanks,
Jonathan

That seems like it'd be trivially handled by compression. I just did a
quick test, and adding "bash, coreutils, " to every single package
record would only add 10k to the entire gzip-compressed file (which is
currently 10816k, so, <0.1%). There are currently ~60k *packages* in the
packages file, so this adds, on average, about a sixth of a byte per
package.

Long-term, I'd like to see that happen. But I'm a huge fan of
incremental steps; defining the problem as "eliminate Essential" makes
it both difficult enough and controversial enough to make it unlikely to
happen at all. Right now, the first step is "let's not let the problem
get any worse, and let's ensure that any new package that might have
otherwise used Essential must instead get packages to add a dependency".

Josh Triplett wrote:
see work in incremental steps, and it's not necessary to get full
consensus for the full plan in advance.  After all, we can learn as we
go and would need to change the plan.

Even so, some *rough* consensus on the plan is very useful for helping
people evaluate that first step.  It also gives people confidence that
it can still move forward even if some people involved no longer have
time to carry it forward themselves.  In this example, I think we
agree about the ideal first (freeze the current Essential set) and
last (eliminate Essential) steps but we don't have a clear picture of
what happens in between.  That makes it hard to be comfortable with
the first step because there's no reason to be confident that we'll
get any further than that.

If we don't have a rough plan, then the current state already seems
okay: whenever someone proposes adding new Essential functionality, we
can discuss it on debian-devel and the most likely outcome is that it
gets rejected.

Some next steps that would make me more comfortable with an explicit
freeze on Essential:

- document how to mark a package as "not safe in normal circumstances
  to uninstall" (such as apt and systemd-sysv).  Is "Priority:
  important" enough or should one use more?  As a package maintainer,
  how do I mark a package as something that a user would never want to
  deinstall in normal circumstances?

- some rough idea about how to find undeclared dependencies.  Do we
  do an NMU to add currently Essential packages as Pre-Depends on all
  packages and then unmark them as Essential and let package
  maintainers sort it out?  Is there some autodetection we can do
  using autopkgtest?

- some rough idea about what to do with packages from outside the
  Debian archive.  Do we provide a way to declare implicit Pre-Depends
  for everything from a particular archive?

To be clear, I'm not saying we need to commit to a plan (and I'm
certainly not saying it should be documented in policy), but a rough
sense of whether this seems possible and whether there are people
interested in moving it forward would help build consensus for the
first step.

Thanks,
Jonathan

Here is a rough plan:

   1. Policy: Packages should declare all their dependencies, even
      essential ones.
   2. Make easier this task: document dependencies, add Lintian checks,
      etc.
   3. Policy: Packages must declare all their dependencies.
   4. Wait until previous Debian releases become unsupported.
   5. Drop support for Essential.

Here is a rough plan:

   1. Policy: Packages should declare all their dependencies, even
      essential ones.
   2. Make easier this task: document dependencies, add Lintian checks,
      etc.
   3. Policy: Packages must declare all their dependencies.
   4. Wait until previous Debian releases become unsupported.
   5. Drop support for Essential.

    Josh> Long-term, I'd like to see that happen. But I'm a huge fan of
    Josh> incremental steps; defining the problem as "eliminate
    Josh> Essential" makes it both difficult enough and controversial
    Josh> enough to make it unlikely to happen at all. Right now, the
    Josh> first step is "let's not let the problem get any worse, and
    Josh> let's ensure that any new package that might have otherwise
    Josh> used Essential must instead get packages to add a dependency".

Josh, my current reading is that there is not support for even the first
step.
I believe Guillem and I have disagreed, and I haven't noticed support
from anyone other than you.

Is there support I'm failing to remember?

I would not attempt to summarize Guillem's concerns.
My concerns are roughly that I think

1) debian-devel consensus is an
adequate block for things getting worse unless there is a good reason

2) I am not convinced that we would (or should) decline to use this
particular hammer if it really is the best tool we have available for a
bind we find ourselves in; nor do I think policy would actually bar us
if we had necessity

3) The benefit I perceive in spending more time trying to figure out
whether I could be convinced that there are no circumstances under which
I'd support a new essential package is less than  what I think we'd get
out of it , so I'd rather not spend the time.

In the interest of being constructive:

A) I do support reducing the essential set over time

B) I would support better education of the community about why we should
be hesitant to support essential: yes on debian-devel

C) I'd support non-normative documentation that we don't expect to
approve new essential packages in the future in policy.

Hello,

Speaking as Policy Editor, I agree.  I don't see any sort of consensus
that we should deprive ourselves of the ability to declare packages
Essential.

This sounds like a good idea to me too.

Worthless documentation, I think.

Fine, then you should choose one essential package and remove it from
the set for bullseye or bookworm.

Worthless documentation, I think.

Fine, then you should choose one essential package and remove it from
the set for bullseye or bookworm.

Javier Serrano Polo wrote:

More specifically, it's the right first three steps.

https://www.debian.org/doc/debian-policy/ch-binary.html#dependencies
currently says

	Packages are not required to declare any dependencies they
	have on other packages which are marked Essential (see below),
	and should not do so unless they depend on a particular
	version of that package.[4]

	[4] [...] If packages add unnecessary dependencies on packages
	in this set, the chances that there will be an unresolvable
	dependency loop caused by forcing these Essential packages to
	be configured first before they need to be is greatly
	increased.

I'd propose that as a first step we change that to

	Packages are not required to declare any dependencies they
	have on other packages which are marked Essential (see below),
	but are permitted to do so even if they do not depend on a
	particular version of that package.[4]

	[4] Functionality is rarely ever removed from the Essential
	set, but packages have been removed from the Essential set
	when the functionality moved to a different package. So when
	depending on these packages for such functionality, it is
	recommended to depend on an appropriate virtual package
	instead.

	Future versions of Debian policy may encourage and then
	require including explicit dependencies even for essential
	functionality. This is helpful to users putting together
	ultra-minimal systems (though Debian does not support omitting
	Essential functionality out of the box), and it means less
	work is required in case the moment comes to remove some
	functionality from the Essential set in the future.

(The next two steps would be to change that from "are permitted" to
"should", and then to "must".)

What do you think?

Thanks,
Jonathan

[4] "Essential is needed in part to avoid unresolvable dependency loops
on upgrade. If packages add unnecessary dependencies on packages in
this set, the chances that there will be an unresolvable dependency
loop caused by forcing these Essential packages to be configured first
before they need to be is greatly increased. It also increases the
chances that frontends will be unable to calculate an upgrade path,
even if one exists.

"Also, functionality is rarely ever removed from the Essential set, but
packages have been removed from the Essential set when the
functionality moved to a different package. So depending on these
packages just in case they stop being essential does way more harm
than good."

Javier Serrano Polo wrote:

More specifically, it's the right first three steps.

https://www.debian.org/doc/debian-policy/ch-binary.html#dependencies
currently says

	Packages are not required to declare any dependencies they
	have on other packages which are marked Essential (see below),
	and should not do so unless they depend on a particular
	version of that package.[4]

	[4] [...] If packages add unnecessary dependencies on packages
	in this set, the chances that there will be an unresolvable
	dependency loop caused by forcing these Essential packages to
	be configured first before they need to be is greatly
	increased.

I'd propose that as a first step we change that to

	Packages are not required to declare any dependencies they
	have on other packages which are marked Essential (see below),
	but are permitted to do so even if they do not depend on a
	particular version of that package.[4]

	[4] Functionality is rarely ever removed from the Essential
	set, but packages have been removed from the Essential set
	when the functionality moved to a different package. So when
	depending on these packages for such functionality, it is
	recommended to depend on an appropriate virtual package
	instead.

	Future versions of Debian policy may encourage and then
	require including explicit dependencies even for essential
	functionality. This is helpful to users putting together
	ultra-minimal systems (though Debian does not support omitting
	Essential functionality out of the box), and it means less
	work is required in case the moment comes to remove some
	functionality from the Essential set in the future.

(The next two steps would be to change that from "are permitted" to
"should", and then to "must".)

What do you think?

Thanks,
Jonathan

[4] "Essential is needed in part to avoid unresolvable dependency loops
on upgrade. If packages add unnecessary dependencies on packages in
this set, the chances that there will be an unresolvable dependency
loop caused by forcing these Essential packages to be configured first
before they need to be is greatly increased. It also increases the
chances that frontends will be unable to calculate an upgrade path,
even if one exists.

"Also, functionality is rarely ever removed from the Essential set, but
packages have been removed from the Essential set when the
functionality moved to a different package. So depending on these
packages just in case they stop being essential does way more harm
than good."

That sounds great to me. This would mean that there'd be no policy
violation involved if someone wanted to send out patches working towards
making a package non-Essential. It's a good incremental step.

That sounds great to me. This would mean that there'd be no policy
violation involved if someone wanted to send out patches working towards
making a package non-Essential. It's a good incremental step.

That sounds great to me. This would mean that there'd be no policy
violation involved if someone wanted to send out patches working towards
making a package non-Essential. It's a good incremental step.

This is very dangerous with respect to upgrade between stable releases.
The issue is at the time a package is made for a stable release, the
state of Debian and Essential: yes packages is not known. It is
unrealistic to expect Debian to plan so far in advance.
Requiring changes to Essential packages to take into account spurious
dependencies is too fragile.

Cheers,

This is very dangerous with respect to upgrade between stable releases.
The issue is at the time a package is made for a stable release, the
state of Debian and Essential: yes packages is not known. It is
unrealistic to expect Debian to plan so far in advance.
Requiring changes to Essential packages to take into account spurious
dependencies is too fragile.

Cheers,

    >> I'd propose that as a first step we change that to
    >>
    >> Packages are not required to declare any dependencies they have
    >> on other packages which are marked Essential (see below), but are
    >> permitted to do so even if they do not depend on a particular
    >> version of that package.[4]

    Bill> This is very dangerous with respect to upgrade between stable
    Bill> releases.  The issue is at the time a package is made for a
    Bill> stable release, the state of Debian and Essential: yes
    Bill> packages is not known. It is unrealistic to expect Debian to
    Bill> plan so far in advance.  Requiring changes to Essential
    Bill> packages to take into account spurious dependencies is too
    Bill> fragile.

I'm sorry, but I consider myself reasonably knowledgable about package
dependencies and Debian--not admittedly as knowledgable as you, but
knowledgable enough that I ought to be able to follow this discussion.
And after spending a couple minutes thinking about the above I don't
understand what you are getting at.

So, please explain in enough detail that we are able to understand and
evaluate your concern.

    >> I'd propose that as a first step we change that to
    >>
    >> Packages are not required to declare any dependencies they have
    >> on other packages which are marked Essential (see below), but are
    >> permitted to do so even if they do not depend on a particular
    >> version of that package.[4]

    Bill> This is very dangerous with respect to upgrade between stable
    Bill> releases.  The issue is at the time a package is made for a
    Bill> stable release, the state of Debian and Essential: yes
    Bill> packages is not known. It is unrealistic to expect Debian to
    Bill> plan so far in advance.  Requiring changes to Essential
    Bill> packages to take into account spurious dependencies is too
    Bill> fragile.

I'm sorry, but I consider myself reasonably knowledgable about package
dependencies and Debian--not admittedly as knowledgable as you, but
knowledgable enough that I ought to be able to follow this discussion.
And after spending a couple minutes thinking about the above I don't
understand what you are getting at.

So, please explain in enough detail that we are able to understand and
evaluate your concern.

control: retitle -1 Permit packages to declare dependencies on Essential packages

Hello Josh,

Could I ask you to explain your wanting to reduce the Essential set for
the sake of small installation size in more detail, including some
numbers, please?  It would be good to get to the bottom of Bill's worry
about this change, but in addition, I would like to see a stronger
positive case.

As someone who is not concerned about small installation size, I rather
enjoy how the functionality of the Essential set isn't likely to be
reduced any time soon.  As an example, I understand that RedHat systems
aren't guaranteed to have Perl on them anymore; I appreciate how when
I'm working with Debian systems, I know I'm not going to have to spend
time improving my knowledge of awk, and anyway having to use a tool
which I think is worse.

I don't mean to suggest that this usecase of mine is decisive, but it
illustrates that the benefits of keeping Essential as it is are clear,
whereas the benefits of reducing Essential are, currently, vague.  Could
we actually decrease installation size in a way that actually benefits
actually existing usecases if we were to start trying to reduce
Essential?  I would like to see evidence of that.

This could refer to concerns with two opposite situations, I guess:

  - When adding new Essential:yes marks, with omitted dependencies (but
    that is already a problem with the current text). In that case
    the dependencies on the new Essential:yes should only be dropped
    after the next stable release, or packages might assume the
    presence of functionality that might not be there.
  - When removing old Essential:yes marks, with adding "excess"
    dependencies, which could possibly introduce dependency loops,
    that could break ordering assumptions in maintainer script
    execution. This can be problematic if the requirements for the
    behavior of an Essential:yes package for the affected package are
    dropped before the next stable release (work even when unpacked,
    Pre-Depends usage, etc), but otherwise it should be fine I guess.

Thanks,
Guillem

This could refer to concerns with two opposite situations, I guess:

  - When adding new Essential:yes marks, with omitted dependencies (but
    that is already a problem with the current text). In that case
    the dependencies on the new Essential:yes should only be dropped
    after the next stable release, or packages might assume the
    presence of functionality that might not be there.
  - When removing old Essential:yes marks, with adding "excess"
    dependencies, which could possibly introduce dependency loops,
    that could break ordering assumptions in maintainer script
    execution. This can be problematic if the requirements for the
    behavior of an Essential:yes package for the affected package are
    dropped before the next stable release (work even when unpacked,
    Pre-Depends usage, etc), but otherwise it should be fine I guess.

Thanks,
Guillem

I'm in full support of trying to reduce the essential set, and this is
something we have already been trying to do for a while now:

https://wiki.debian.org/Proposals/EssentialOnDiet
https://wiki.debian.org/BusterPriorityRequalification

I also do not see the need to disallow adding new stuff to it if need
be. I think there's enough resistance in place that proposing that on
debian-devel would be met with quite some skepticism, *but* in case it
is needed, it would seem odd to do it anyway, even if policy disallows
it.

While the idea of completely getting rid of the Essential:yes concept
has been floated around in the bootstrapping and multiarch circles,
and seems like an interesting idea, I think it currently has too many
problems to be entertained.

Several of the Essential:yes properties [P] would need to be preserved
somehow for the packaging system to be able to operate as of now. Making
that marking implicit (some kind of lore) seems worse, than the current
explicit one, because the Essential field does not get involved much in
the package system operations anyway, it's more of a contract to be
honored by *maintainers*! Having to add Pre-Depends to tons of packages,
would probably substantially strain our dependency resolvers. Maintainer
scripts in at least the essential set are another problematic area,
so they would need to be reduced substantially, or eliminated
completely [B][D].

 [P] <https://lists.debian.org/debian-dpkg/2020/03/msg00002.html>
 [B] <https://wiki.debian.org/Teams/Dpkg/Spec/InstallBootstrap>
 [D] <https://wiki.debian.org/Teams/Dpkg/Spec/DeclarativePackaging>

I think there are some rough plans for some of the problematic pieces,
but otherwise this seems rather premature, and I concur that the
current state seems fine.

This is already supported with the Protected (old Important) fields,
and was implemented precisely to make it possible to split the
essential set into its different facets.

  <https://wiki.debian.org/Teams/Dpkg/Spec/ProtectedField>

Not all packages need Pre-Depends on currently Essential:yes packages,
but this would require careful consideration. But as mentioned above,
having to increase the amount of Pre-Depends relationships could make
our dependency resolvers and upgrades unhappy.

Trying to detect dependencies is going to be tricky, given that most
of the essential set are callable interfaces, which need to be checked
within scripts calling those tools, or compiled code doing that while
possibly composing tools names are run-time, etc. Using a full code
search would be the usual approach, but that might not be scalable
depending on the dependency.

I don't think we usually take external packages into account TBH,
besides letting our usual one stable release gap in between.

Making it possible (or more explicit) in policy to remove packages from
the essential set might be worthwhile by for example at least covering
that some of the requirements do not need to be met while those removal
transitions are taking place, and that those transitions need to be
discussed in debian-devel, but then we have already been doing that
w/o much of a fuss anyway, so…

Thanks,
Guillem

I'm not sure about Josh, but I think the main reasons for wanting to
reduce the essential set are:

  - Making chroots/containers slimmer, which can have a substantial
    impact when needing lots of them, where even few MiB can make a
    difference.
  - Making bootstrapping (build and installation) in general easier,
    even though for the former these packages also need to then
    be ideally removed from the build-essential set too.

Then there are these:

  - Making the packaging system less opaque, as the full implications
    of Essential:yes do not seem generally clear to all maintainers?
  - Making it easier to use alternative implementations for some of
    these packages, as then it's easier to know what depends on what,
    instead of having to unconditionally provide the entire interface
    surface for anything.
  - Making it easier to handle some multiarch problems, as then
    dependencies can be explicitly marked as needed.

Thanks,
Guillem

Hello,

Thank you for this, but I was hoping for some more specifics.  For
example, what are some examples of large Essential: yes packages that
might actually, in practice, be removable?

Hi,

Sean Whitton wrote:
e2fsprogs, which is ~2.1 MiB.  (You might not consider that large, but
when you multiply that by "every Debian installation everywhere", it
adds up.)

https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=nonessentiale2fsprogs;users=helmutg@debian.org
shows that people are already adding explicit dependencies on it,
which means that
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=954794;msg=111 is
the de facto policy / what people believe policy to say.  (Which is a
surprise to me, but it's a useful signal.)

Thanks,
Jonathan

So, should policy change? Currently, it says that packages "should not"
declare essential dependencies, which is more permissive than "must
not". Is the workflow with e2fsprogs the path to follow?

So, should policy change? Currently, it says that packages "should not"
declare essential dependencies, which is more permissive than "must
not". Is the workflow with e2fsprogs the path to follow?

Hello,

A lot of systems do not actually need *anything* from util-linux.
Yet is is Essential: yes, and because of the large set of programs
in it, removing the Essential flag on it will be extremely painful.

Even more so when other packages cannot depend on it for an orderly
transition.

~5MB installed, give or take for libraries which may also become
removable.

Chris

Hello,

Thanks. Clearly this is a situation that we want to avoid happening
again.

I don't see how the current text would be inadequate, though.  For a
package like util-linux, a discussion on debian-devel would result in a
resounding "no".