Debian only supports merged-usr since Bookworm. We should update policy
to reference /usr/bin/sh and similar paths to describe recommended
shebangs for scripts.

I heard many times the policy maintainers mention something along the
lines of 'policy should not be a hammer to beat other maintainers
with'. Today I saw policy being used to force a maintainer to re-add
support for the deprecated and unsupported split-usr filesystem layout,
as 'policy only mentions /bin/sh, not /usr/bin/sh'.

So let's update the policy to refer to modern and supported filesystem
paths as adopted by Debian de-facto and de-jure, and stop other
maintainers from getting beaten with it.

Patch attached and also pushed to
https://salsa.debian.org/bluca/policy/-/tree/bin_sh

Unsubscribe:

 e.little598@gmail.com

    Luca> Debian only supports merged-usr since Bookworm. We should
    Luca> update policy to reference /usr/bin/sh and similar paths to
    Luca> describe recommended shebangs for scripts.

I do not support this change.  /bin/sh should still be the recommended
interface path for the posix shell.  Among other reasons, it promotes
compatibility between Linux and other posix architectures.  Besides
that, I suspect there are cases where tools look for /bin/sh in a #!
line and do not accept /usr/bin/sh.

I absolutely agree that the physical path of the posix shell should be
/usr/bin/sh.
I disagree that we should be changing interface paths for items
traditionally found in /bin.

    Luca> Debian only supports merged-usr since Bookworm. We should
    Luca> update policy to reference /usr/bin/sh and similar paths to
    Luca> describe recommended shebangs for scripts.

I do not support this change.  /bin/sh should still be the recommended
interface path for the posix shell.  Among other reasons, it promotes
compatibility between Linux and other posix architectures.  Besides
that, I suspect there are cases where tools look for /bin/sh in a #!
line and do not accept /usr/bin/sh.

I absolutely agree that the physical path of the posix shell should be
/usr/bin/sh.
I disagree that we should be changing interface paths for items
traditionally found in /bin.

/bin/sh is not universally compatible with non-Linux OSes.

Also I thought that policy should not be used to beat other developers
(it is because of this) and it should reflect the common practices
adopted in Debian (it does not because of this). Is that no longer the
case?

    Luca> /bin/sh is not universally compatible with non-Linux OSes.

I claim it is more compatible.


    Luca> Also I thought that policy should not be used to beat other
    Luca> developers (it is because of this) and it should reflect the
    Luca> common practices adopted in Debian (it does not because of
    Luca> this). Is that no longer the case?

I'd consider it a non-RC bug if someone were manually writing
#!/usr/bin/sh

We can debate about normal vs minor.

I do not think it should be a bug if some automated build process found
/usr/bin/sh and stuck that into a script.
I'd support a policy change to make that clear.

How would such a change look like?

    Luca> How would such a change look like?

I looked at your patch.

In most of the cases you are changing non-normative language.
That is, parts of policy that do not create a requirement.
For example:

That creates no requirement on a package.

I.E. in the cases you adjust, I think it is already
not a bug, and it would be inappropriate to use existing policy language
to complain about which interpreter path people use.

There is one case however where I think your patch adjusts normative
language.

I propose the following adjustment to your patch.  Rather than mandating
a particular path for a csh interpreter, make it clear that the policy
requirement is that the csh script start with a #! line rather than
simply assuming csh will interpret the script as was all too common back
in the days of csh:

@@ -266,7 +266,7 @@ including ``open``, ``print``, ``close``, ``rename`` and ``system``.
 Programming Considered Harmful*, one of the ``comp.unix.*`` FAQs, which
 can be found at http://www.faqs.org/faqs/unix-faq/shell/csh-whynot/. If
 an upstream package comes with ``csh`` scripts then you must make sure
-that they start with ``#!/bin/csh`` and make your package depend on the
+that they start with the path of a csh interpreter such as ``#!/bin/csh`` and make your package depend on the
 ``c-shell`` virtual package.

 Any scripts which create files in world-writeable directories (e.g., in

I think the other hunks of your patch are unnecessary to avoid policy be
used to create bugs in this space.

In practice though they are used as normative, for example Lintian
raises an error (not even a warning, an actual error). So given they
are not literally normative, wouldn't it be fine to change them, or
alternatively drop any specific path mention? That way there is no
doubt that we can change Lintian and close such bugs. Right now it's
not that clear at all, as evidenced by Lintian and these bugs.

How about:

+that they start with the path of a csh interpreter such as
``#!/bin/csh`` or other valid paths, and make your package depend on
the

As long as our official buildds are non-merged-/usr, I would consider use
of that path in scripts that get run at package build time to be
potentially RC, in fact. I have tried to arrange for our official buildds
for suites >= trixie to become merged-/usr (#1050868, #1025708) but that
work is awaiting review.

Until that change lands, or our official buildds for
trixie/sid/experimental become merged-/usr some other way, encouraging
maintainers to use interpreters that were historically in /bin via their
/usr/bin paths is actively harmful and we should not do it. Please don't
make the transition to merged-/usr-everywhere more contentious than it
already is.

Hard-coding "#!/bin/sh" is not compatible with every operating system,
but it's much more widely compatible than hard-coding "#!/usr/bin/sh",
particularly for scripts that are meant to be backportable to older
versions of LTS distributions (notably Debian!). Even NixOS, an operating
system that is notable for the extent to which it intentionally avoids
using FHS paths, has made the pragmatic choice to provide /bin/sh.

I would not want to encourage Debian maintainers to use
"#!/usr/bin/env sh" for shell scripts, even though it's more broadly
portable than assuming that /bin/sh is a POSIX shell.
This is analogous to how we encourage (and sometimes require) Debian
maintainers to use "#!/usr/bin/perl" and "#!/usr/bin/python3" in
preference to indirection via "#!/usr/bin/env perl" or similar, even
if their upstream uses "#!/usr/bin/env perl" for wider portability.

We certainly shouldn't require shell scripts in Debian to go through
the same contortions as Autoconf-generated configure scripts to find
a POSIX shell (see `info autoconf 'Portable shell'`) because that's
unnecessary runtime and maintenance overhead on any GNU/Linux system,
any of the non-Linux ports that Debian has attempted to support in the
past, and hopefully any free or proprietary Unix that is still relevant
to Free Software authors this decade.

    smcv

In general, if Lintian is diagnosing paths as an error when they are
allowed by Policy (therefore not a serious bug) and work in practice
on all relevant systems (therefore not a grave bug), then the place to
change that would be Lintian, not Policy.

However, because our official buildds for trixie/sid/experimental
are still using the transitional non-merged-/usr layout at the time
of writing, it is still valuable for Lintian to care about whether
interpreters are specified with their /usr path or not: so I think Lintian
is still entirely correct to encourage #!/bin/sh over #!/usr/bin/sh,
and so on.

Also, the same version of Lintian is used to check packages targeting
any supported suite (including bookworm-backports, bookworm, and even
bullseye), and Lintian does not always have enough information to know
which suite is being targeted (consider UNRELEASED packages). bookworm
buildds are likely to remain non-merged-/usr indefinitely, to ensure we
have a working upgrade path from bullseye, so using the non-traditional
path to an interpreter (for example /usr/bin/sh or /bin/perl) is in
practice still RC for bookworm if the script might be invoked during
build. So I think it would be premature to change Lintian for this,
even after our official buildds for suites >= trixie become merged-/usr.

Using the non-traditional path to an interpreter is also *definitely*
RC for *bullseye*, because bullseye systems that were upgraded from
a sufficiently old release are non-merged-/usr unless the user has
specifically taken steps to merge it.

I think it would be appropriate to change Lintian to accept /bin/sh
and /usr/bin/sh as interchangeable after trixie is released, at which
point bullseye-backports will have been discontinued, bullseye will be
LTS-maintained, and bookworm-backports will only be receiving targeted
stable-release-suitable changes which are unlikely to regress the choice
of interpreter path.

    smcv

I am aware, and we are targeting 12.2 for that (and I hope we manage
to hit the target). On the other hand, changing Lintian takes several
months usually. Policy is even slower, and moves at glacier speed, so
when I opened this yesterday I did so with the intention that by
starting now, we might get a chance to have it sorted in time for
Debian 14 (Forky is it?). Further delaying means even that target
might be missed.
Fixing the current loophole in usr-is-merged that is being used to
abuse maintainers is imperative and high priority, so I hope we will
sort out the buildd situation within a month if all goes well, and in
a couple of months at worst. I fully expect this to happen before
anybody looks at the Lintian MR, and long long before this policy
change has a chance to go anywhere.

The intention here was not to encourage to change or switch, but to
stop bug reporters from using policy to beat maintainers. Given I am
told these sections are not normative, I am perfectly fine with
rewording in any way that achieves that objective - ie, mentioning
both paths, or none, for example.

As above, the intention here is not to encourage one way or the other,
but to take the hammer away. These wordings are not normative anyway,
but they are used as if they were, and I think that's something we
want to fix regardless.

Why should that matter to Debian?

Should Debian invest resources to make upstream software more portable
to systems like Mac OS X? For me this seems out of scope for Debian.

Should we require Debian maintainers to patch this in software packaged
for Debian? Why?

I think it is fine to wait with this until buildds use merged-/usr for
builds, but I don't see any reason to spend resources on this after
that.


Ansgar

    Ansgar> On Wed, 2023-09-06 at 16:51 -0600, Sam Hartman wrote:
    >> > > > > > "Luca" == Luca Boccassi <bluca@debian.org> writes:    
    >> Luca> /bin/sh is not universally compatible with non-Linux OSes.
    >>
    >> I claim it is more compatible.

    Ansgar> Why should that matter to Debian?


Debian has traditionally valued supporting common interfaces like posix,
fhs, Linux ABIs, etc where that makes sense.
We recently had a discussion of the value of interfaces in  the
discussion of changing the ABI to make merged /usr easier, and I do not
want to revisit that.

I do value this sort of interface stability, and Debian's alignment with
my values is one of the things that drew me to Debian.

So, yes, I do believe we should support encouraging portability where
that is reasonable for us.
I admit that I care more about OSes like FreeBSD than Mac OS.

More over, when merged /usr was presented to the project, it was
presented as a way to move the physical locations on files and as a way
to create an alias so that we didn't need to argue when different
distributions  made different decisions about /bin vs /usr/bin.
It was not presented as a change to common interface paths like /bin/sh.

This request is new, and given the politics, is something I find highly
problematic.
It is not abusing maintainers to ask them to respect long-standing
interfaces like the location of /bin/sh.
As Simon has pointed out, in a number of cases it is still actually RC
because it can break builds.
It is not abusive to ask maintainers to fix issues that prevent their
packages from building.
We make mistakes.
It is not abusive to get the severity of a bug wrong or even to disagree
with the severity of a bug.

I am sympathetic to the idea that after buildds are updated, we we might
want to reduce the severity of not using longstanding interface paths,
and in some cases not even treat it as a bug.
I reject the idea that /usr/bin/sh should be preferred over /bin/sh or
even the idea that it should be equally preferred.
I am open to the idea that we may not care to record that as a bug or
spend the time fixing it.

However, the tone and approach in this discussion does not encourage me
to participate.
If the current tone continues, I will use up the energy I have for
working toward a compromise and simply stand behind my support of
longstanding practice and  support of portable interfaces.

I would. Having two paths for the same thing is a technical debt going forward.

Cheers,

    Bill> I would. Having two paths for the same thing is a technical
    Bill> debt going forward.

I think the TC has made it clear we're committed to usrmerge at this
point, and I think that one of the drivers behind usrmerge is that we
gain more from having both /bin/python and /usr/bin/python work than we
lose by having two paths.
I understand people disagree,
but I think we should fall in behind the TC decision and accept we've
decided that in this instance we value aliasing.

Hi Luca,

I disagree. The promise of merged-/usr has always been that both paths
are valid. /bin/sh remains the location recommended by external
standards and (like the dynamic loader path) should remain the way it
is.

This can also be addressed by adding a note to policy that allows
maintainers to rely on the aliasing. If there was a need to refer to the
shell via /usr/bin/sh, we would aim for eventually removing the aliasing
symlinks. That's not what we're up to.

I don't think this is right. We intend to finalize the /usr-merge
transition by moving files from / to /usr. This is is an implementation
strategy that arises from the constraints set by the current
implementation of dpkg and other components. It is not a new filesystem
layout that we expect upstreams to support. Rather, we promised to
upstreams that both ways will work. The aspect that in a data.tar we'll
have to install files to /usr is a technical one and can be supported by
debhelper. Still, packages may assume that referencing files they
installed to /usr via aliased paths in / will continue to work.

Nack to this particular change, but I agree that it is worth considering
two changes to policy sooner and later:
 * Making it explicit that referring to files via either paths for
   read-only consumption is ok.
 * DEP17 aims for not installing any files in aliased locations and we
   should encode that in policy once there is wide adoption of this rule
   in binary packages.

Would you agree to repurpose this bug to propose the former change?
While my variant is weaker, it still prevents people from using policy
to require supporting split-/usr.

Helmut

Yes, that is fine by me, as explained in later replies my main
intention is to fix the issue that some wording is being used to
reintroduce things that should not be reintroduced

If I understand you correctly, "Reintroduc[ing] things that should not be
reintroduced" means not acting on bug reports where someone says
"Please change #!/usr/bin/sh back to #!/bin/sh".

If I'm understanding the technical question correctly, "#!/bin/sh"
works for both merged-/usr and split-/usr, while "#!/usr/bin/sh" works
only for merged-/usr and breaks for split-/usr.

Now, how to handle this situation? In my naïve point of view and a
bit late at night I see two options for maintainers. They can say …

1) "Well, right, *sigh*, ok, let's take the five minutes to change
   this back to "#!/bin/sh", and everyone's happy."

… or …

2) "No way, we are usr-merge, resistence is futile, go <expletive
   deleted> yourself, "#!/usr/bin/sh" is the default, bad luck for
   you !!!!111"


I'm a bit sad that many discussions in Debian (like this one) turn
into the "my way or the highway" lane, and I'd rather see a way of
cooperation which gives more leeway to others and take small extra
steps to accomodate minorities, when it doesn't have any actual cost.


Cheers,
gregor

The entire point of the decision that applies since bookworm is that
there is only one supported form. That's not an accident, it is the
entire point of the whole thing. As in, there is no such thing as
"breaks split-/usr" because split-usr is no longer a thing that exists
in Debian stable/backports/testing/unstable/experimental.

And I am more than a bit sad that sensible, clear-cut, binding and
already-implemented decisions taken by our constitutional bodies get
constantly second-guessed and belittled because of an irrational
attachment to inconsequential accidents of history. But what can we
do, we'll just have to be sad together, I guess.

Aside from that, in the future please avoid using the word
'minorities' when talking about silly things such as liking or
disliking symlinks, as in common English it is used to refer to much
more serious matters.

Luca Boccassi <bluca@debian.org> writes:
Nothing about this discussion style helps in making a good decision.

Luca, you have asked repeatedly what you can do to advance the status of
your open Policy bugs.  Most of the answer to that question is that a lot
of it is outside of your control; I've been having a spectacularly
annoying summer in a bunch of minor ways and, frankly, simply haven't done
much of anything I'm responsible for in Debian.  But the most helpful
thing that you could possibly do would be to stop being utterly
emotionally exhausting to deal with.

Case in point: now I'm writing this email message rather than doing any of
the other things that I am quite sure you would rather that I be doing and
that you would consider a much higher priority.

In this bug, about a point that from the Policy perspective is mostly not
even normative, which appears to partly be a bug report against Lintian by
proxy (the Policy Editors do not maintain Lintian), you started at an
emotional level of eight out of ten before anyone even disagreed with you.
Let me quote the start of this bug:

    I heard many times the policy maintainers mention something along the
    lines of 'policy should not be a hammer to beat other maintainers
    with'. Today I saw policy being used to force a maintainer to re-add
    support for the deprecated and unsupported split-usr filesystem
    layout, as 'policy only mentions /bin/sh, not /usr/bin/sh'.

    So let's update the policy to refer to modern and supported filesystem
    paths as adopted by Debian de-facto and de-jure, and stop other
    maintainers from getting beaten with it.

So right from the first message, you're saying that the goal of this
proposal is to stop other people from doing something socially you don't
like that Policy doesn't even tell them to do.  And you're using phrasing
that you know is loaded and contentious and that you know the people
you've been arguing with all over Debian will disagree with.

I have no idea why.  Because you like making them mad?  Because you're mad
and you want to keep rubbing in their face the fact you disagree with
them?  Maybe you think this is your strongest argument?  (It is not.)

This is pretty much guaranteed to turn the bug into a fight regardless of
its technical merits, because this presentation essentially says that
you're spoiling for a fight and want someone to attack you.

I cannot remember the last time I have seen someone who is this unwilling
to win an argument.  It's driving me nuts.  You are winning your technical
arguments in Debian!  The technical design that you support is being
implemented because you and others have convinced people that it is a good
idea, even though some people are very upset about this!  There are even
people who don't entirely agree with you investing significant amounts of
time and energy to help you realize your technical vision of how Debian
should be assembled.

And yet, every time I see you in a Debian architectural discussion, you're
fighting with everyone in sight as if you're losing.  You come across as
the most furious winner of an argument I can remember seeing.  This is
directly undermining what you're attempting to achieve, because the sheer
vehemence of the way that you argue every point you care about means that
even people who agree with you don't want to help you get things done
because even being around this level of fighting is deeply exhausting.

A lot of this hasn't been on Policy, although you do have some of the same
tendency to reply to everything you disagree with until people stop
disagreeing even here.  But elsewhere, including the Technical Committee
discussion that I assume you were referring to above, it happens a lot.  I
can only assume that you think you have to shove harder and harder to get
things to move faster, but this isn't how anything works, particularly in
a volunteer organization.  People work on things they enjoy working on.
Calm and productive technical work is enjoyable to a lot of us.  Watching
people fight is not enjoyable.  What you're actually accomplishing is
building a subconscious emotional association in the brains of people who
are watching and reading between your name and angry arguments.  You are
literally training people to avoid you.

You are never going to convince people to stop disagreeing with you.
Never.  We have a process for making decisions because sometimes consensus
is not achievable, and the people who don't agree with those decisions
may, in fact, *never* agree with those decisions.  They get to do that.
It's not a crime.  It's also not a personal offense against you.

When people voice that disagreement, again, to a decision-making body and
the immediate response is for multiple members of that body to politely
but quite clearly indicate that, although they're willing to listen, the
discussion is probably going nowhere and no decisions will change, this is
what winning an argument looks like.  This is what you're going to get.
You're not going to get unanimous consensus.  Take the win and *leave it
alone*.

Right now, you keep risking undermining architectural decisions that
you've worked hard to achieve because you pile into discussions with your
intensity knob already maxed and say a whole lot of confrontational stuff,
some of which is careless or imprecise, and then the people *who already
agree with you* feel obligated to disagree and clarify.  And *then you
start fighting them*, and absolutely nothing about this is achieving any
goal that you have.

Please, for the sake of my blood pressure, I beg of you, dial it *way
down*.  I swear that when I do process Policy bugs I will read all the
messages and understand the technical arguments and clearly state what I
think the most convincing points are, and everyone will get a chance to
review that, and changes can even be reverted later if we get it wrong.
Nothing is going to happen by surprise, and you do not have to be the last
arguer standing in order to get your change into Policy.  The more
messages there are, and the more emotional heat there is, the more energy
the whole process requires and the longer it takes.

I hope you will fare better in autumn, and take all the time you need.
If I inquire about status on the other open bugs is only because I am
unsure of whether I have missed something on my side, and want to
double check that nobody is waiting on new revisions or so, because I
find it hard to track things in long email threads, and I easily miss
stuff, especially as when I asked if I missed something, someone else
said "Yes, Russ posted some comments on your most recent revision, I
believe.", but I couldn't find it. I wouldn't have thought that
sending a reply with "Anything I can do to unblock this?" once a month
or so would be classed as abusive, but given it exhausts you, I'll
stop and just hope I got everything right as-is.

No. The point of this proposal is to fix something that is
self-evidently unclear - non-normative (as I've been told, I thought
it was normative when I opened the bug) language being used as if it
were normative, both by Lintian (yes, the issue has been raised
separately and a fix provided) and by bug reporters. Given, again,
I've been told many times (I believe by yourself, too) that policy
should not be used as a tool to bash developers with, and given I have
witnessed precisely that the other day, I hence decided to spend some
of my time to try and fix that.
As already discussed earlier on this thread, I am fine with changing
the wording of the patch however it fits best, as long as it improves
clarity for this purpose.

For a message that started with "We are not going to have a Policy
conversation like this", this sure looks a lot like a long-winded
tangent to me. I don't think this policy issue is the place nor time
for such a flamewar, but I am going to clarify a few points given you
brought them up.

Firstly, I couldn't possibly care less about who agrees or disagrees
on what with whom. Nor who 'wins' or 'loses' - this is not a football
game. If someone writes a blog post saying "Debian is pure shite and
it's all bluca's fault", I'm not even going to flinch, have at it,
disagree all you want. I do however care a great deal when work is
hampered by obstructionism, and misinterpretation of the rules
(whether willful or not we shall not speculate), and all the way up to
open and plain sabotage, because that's not just "disagreement". This
has been and still is a constant struggle. For years. You don't think
_that_ is exhausting? Have you stopped for a second to think that
maybe, just maybe, that's the reason the "intensity knob is already
maxed"?

Secondly, and less importantly, while I appreciate it's not how you
handle policy changes, the way the rest of the distribution works is
by 'building consensus' on mailing lists. Now I don't particularly
like it, but it is what it is. And that means if somebody comes up
with the most egregious nonsense like, to pick a completely fictional
example, "hey folks, usr-merge broke docker, rsync and ansible, we
need to revert it", and it is left unchallenged, then it becomes doxa.
So it has to be challenged. Every time. After half a decade, you don't
think _that_ is exhausting?

    Luca> Secondly, and less importantly, while I appreciate it's not
    Luca> how you handle policy changes, the way the rest of the
    Luca> distribution works is by 'building consensus' on mailing
    Luca> lists. Now I don't particularly like it, but it is what it
    Luca> is. And that means if somebody comes up with the most
    Luca> egregious nonsense like, to pick a completely fictional
    Luca> example, "hey folks, usr-merge broke docker, rsync and
    Luca> ansible, we need to revert it", and it is left unchallenged,
    Luca> then it becomes doxa.  So it has to be challenged. Every
    Luca> time. After half a decade, you don't think _that_ is
    Luca> exhausting?


Thanks for sharing this.
I understand what you've been doing a lot better now.  And if it is
actually true that you need to challenge these assertions every time,
your behavior makes more sense to me.

However, it's been my experience that challenging these assertions every
time is unnecessary.  It actually makes it harder to judge consensus and
keeps discussions alive longer than they need to.
I actually think that there are cases where if you had said less, a
consensus you would have liked just fine would have emerged faster  than
has happened.
Often when someone says something rediculous in a consensus discussion
it is best to let it fall into a void of silence.
Challenging it can sometimes just give it energy.

I would be very open to helping you (or anyone) explore how to work more
efficiently in a consensus environment and how to espace from a
consensus discussion when consensus is the wrong tool.
I realized that I focused a lot on consensus during my term as DPL.  A
lot of that was that I was hoping to help people explore how to approach
consensus discussions better (and because it was the right tool for some
of those discussions).
But there are a lot of discussions where consensus is the wrong tool.
And now that we've leveled up how we approach consensus discussions,
perhaps we should level up how not to have them:-)

Control: retitle -1 debian-policy: stop referring to specific paths in scripts shebang examples
policy
removes the full paths where it is not necessary, as the text simply
talks about the different types of interpreters, and in two occasions
where a shebang is specified, a note is added to mention that other
valid paths are also acceptable (without saying what they are), taking
in both your and Sam's suggestions as I've understood them.

This thus doesn't encourage nor suggest any changes, but simply removes
the ambiguity over what is and is not allowed by norm.

Does this look more amenable?

From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
From: Luca Boccassi <bluca@debian.org>
Date: Wed, 6 Sep 2023 22:41:58 +0100
Subject: [PATCH] Stop referring to specific paths in scripts shebang examples

These are being mistaken for normative languages, but they are just
examples. Remove full paths where not needed, and specify that other
valid paths (without mentioning them explicitly) are also acceptable
in other places.
---
 policy/ch-files.rst | 20 +++++++++++---------
 1 file changed, 11 insertions(+), 9 deletions(-)

diff --git a/policy/ch-files.rst b/policy/ch-files.rst
index b34c183..593d586 100644
--- a/policy/ch-files.rst
+++ b/policy/ch-files.rst
@@ -211,8 +211,8 @@ may instead be easier to check the exit status of commands directly. See
 Every script should use ``set -e`` or check the exit status of *every*
 command.

Simon pointed out that this bug is not yet ready to act on, which was very
helpful.  Thank you.  However, presumably the buildds will be /usr-merged
at some point in the not-too-distant future, and we do need to decide what
to do after that point.

I think the root problem behind this bug is that it is revealing we have
not made a decision about /bin and /usr/bin path references in Debian
after /usr-merge.  Various people, myself included, made assumptions about
what the policy would be, but we never actually decided anything that I am
aware of and people's assumptions are not matching.  I think we need to
talk about this directly, after which what to do with this bug will
probably become obvious.

So far as I can tell, there are three main possibilities:

(a) Although /bin and /usr/bin are merged (and likewise for the other
    merged paths), Debian will continue to require (or at least recommend)
    use of /bin paths for things such as /bin/sh that historically used
    those paths.

(b) Since /bin and /usr/bin (and likewise for the other paths) are merged,
    /bin/sh and /usr/bin/sh are equivalent.  Packages can use whichever
    path they want, and Debian will end up with a mix of both references.

(c) Although /bin and /lib technically work due to the aliasing, they are
    deprecated and everything in Debian should stop using those paths.
    All paths should point to /usr/bin and /usr/lib now.

Although Luca made a few arguments in the direction of (c), my
understanding of his current patch is that it essentially implements (b)
for script interpreters, although without encoding into Policy an explicit
decision between these three options (quite understandably because he was
trying to deal with a narrower issue).  Several other people were, I
think, arguing for (a), but I'm not sure if they would continue to do so
when it's put in these terms.

Policy currently says nothing significant about this (hence most of the
text so-far discussed being informative) because, up until now, (a) was
the de facto policy.  If you didn't follow (a), you would get not-found
errors and your package would have an RC bug because it wouldn't work.

If we do nothing, we'll get (b).  I think that's reasonably obvious, since
there is no technical factor forcing either (a) or (c).  Both paths will
work without any noticable difference, so some people will use /bin and
some people will use /usr/bin.

That said, I would rather make an explicit choice rather than decide by
default, since otherwise this seems like the kind of thing where people
are going to get conflicting advice, which is frustrating for everyone.

If someone wants to argue for (a) or (c), I think the biggest problem with
either of them is an enforcement mechanism.  How are we going to check
whether packages are following the rules?  Lintian and a bunch of grep
patterns is sort of an unsatisfying 90% solution, and people will ignore
it or just not use Lintian.  There is also no technical reason why both
paths will not work, so people are going to get grumpy about being told
what to do and some people will view this as makework.  I think any
proposal to pick (a) or (c) needs to wrestle with that.

I will also say that it will be very hard to convince me that Debian
should give Policy advice like (a) or (c) but not actually enforce it
anywhere.  We have a long history to look at for how those sorts of
mandates go.  Conscientious packagers who read Policy carefully follow the
rules and go to extra effort to do so, but other people will never see
that advice or not pay attention to it.  That means that the effort is
mostly wasted, because no one can rely on the invariant that either (a) or
(c) is attempting to achieve.  I am not a big fan of asking people to do
extra work without some clear benefit from doing that work, which mostly
requires uniformity.

One last point about this decision: although there are a few style
recommendations in it, Policy is not a style guide.  The point of Policy
is to describe the things we should or must do, or at least that the
project as a whole wants to encourage, that have a concrete effect on the
quality of the distribution.  I'm reluctant to add more style advice to
Policy, particularly about things that are not specific to Debian.  If
we're going to require or recommend something, I think we need to have a
concrete goal in mind for what that requirement or recommendation is going
to achieve.

Hi,

As far as I understand people wanting merged-/usr want (b) (I do).
There have been people advocating (a) in this bug.

However, there is a proposal by Jackson for an alternative filesystem
layout based on symlink farms in consideration by the technical
committee.  This advocates removing compat symlinks in /bin, /sbin over
time[1], thus requiring (c).

The technical committee should therefore probably be aware of this
policy issue in their consideration of #1050001; the resolution of
which might also cover this issue (#1051371).

Ansgar

  [1]: https://bugs.debian.org/1050001#33

Russ, there is a quite related point I do not think the TC addressed directly,
but I can easily be mistaken: the default PATH.
It is currently defined in /etc/login.defs (unless my copy of this file is out of date):
as
ENV_SUPATH      PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ENV_PATH        PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

so in pratice, tools like 'which' will always favor /usr/bin over /bin

$ which sh
/usr/bin/sh

One of the issue in the past is that reproducible build was broken because different
build environment lead to different paths. We at least need to address that.

Personnally, I favor keeping /bin/sh for consistency, but that is aside.

Cheers,

    Russ> with a narrower issue).  Several other people were, I think,
    Russ> arguing for (a), but I'm not sure if they would continue to do
    Russ> so when it's put in these terms.

It's hard for me to express what I was advocating for in the terms you
have adopted.
I think you've  advanced the discussion significantly by making it clear
where the difficulty is, which will help me focus my argument.


    Russ> If someone wants to argue for (a) or (c), I think the biggest
    Russ> problem with either of them is an enforcement mechanism.

My problem with (b) is that I value interfaces and that  especially for
/bin/sh I do think that /bin/sh is more portable as an interface path
than /usr/bin/sh.
I care a lot that we use /bin/sh in our documentation and examples
(especially in policy).
I care a lot that we point out that the reality of the situation is
people  will see other paths.
At least for things traditionally in /bin I do not want to encourage
those other paths, but I also don't think it is often a good use of
project resources to "fix" those other paths.
In some cases (for example what version of a path autoconf detects), I
think that patching individual packages to detect a particular path
would be net harmful.

So I want to argue for (a) with no  enforcement mechanism in packages.

1) Policy should encourage /bin paths for binaries traditionally in
/bin. (At a minimum I'd like to see this for /bin/sh and /bin/bash).
That at most makes it a minor bug if you don't follow that
encouragement.

2) The examples in policy should use the standard interface paths.
(This is the thing I care most about).

3) I'd like to see policy point out that /usr/bin paths will end up
being used, and packages SHOULD work regardless of which path is used.

I think there are some complexities here.  Imagine someone writes a tool
to determine if something is a shell script.  If it's security
sensitive, I think it is important that it work both for /bin/sh and
/usr/bin/sh.

If it's a syntax highlighting program and it only detects
/bin/sh as a valid shell script, I'd be okay if the maintainer didn't
want to fix it but instead said "fix your shell scripts if they say
/usr/bin/sh."  (I think the maintainer would be doing a better job if
they supported both).  But I would not be comfortable with a syntax
highlighting tool pushing people toward /usr/bin/sh.

While that could be said for the original revision, in my view that's
not really the case for the latest that I posted?

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1051371#120

So I would prefer if this was a clone rather than a retitle/repurpose.
Unless I'm missing something, the changes linked above should be
uncontroversial and simply remove excessively normative language in
what are essentially examples that should have been taken as such -
and that currently are not. So, could that be taken forward
independently of the problem you define below?

I think b would work out fine, but if we want to start being normative
then it probably would make more sense to go for the new layout rather
than the old. It would seem strange to have lived with the old layout
and no rule, and suddenly add a rule for the old layout just as it has
been phased out, no? But IMHO this is quite low in the priority list,
as you said any of these would work just the same.

To be specific, I needed to add
SHELL=/bin/sh GREP=/bin/grep SED=/bin/sed DD=/bin/dd
to configure to get reproducible build. (This was in December 2018).

Cheers,

    Luca> On Wed, 13 Sept 2023 at 04:48, Russ Allbery <rra@debian.org> wrote:
    >>
    >> Control: retitle -1 Post-/usr-merge paths for script interpreters
    >>
    >> Simon pointed out that this bug is not yet ready to act on, which
    >> was very helpful.  Thank you.  However, presumably the buildds
    >> will be /usr-merged at some point in the not-too-distant future,
    >> and we do need to decide what to do after that point.

    Luca> While that could be said for the original revision, in my view
    Luca> that's not really the case for the latest that I posted?

    Luca> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1051371#120

    Luca> So I would prefer if this was a clone rather than a
    Luca> retitle/repurpose.  Unless I'm missing something, the changes
    Luca> linked above should be uncontroversial and simply remove
    Luca> excessively normative language in what are essentially
    Luca> examples that should have been taken as such - and that
    Luca> currently are not. So, could that be taken forward
    Luca> independently of the problem you define below?

I agree the above is uncontroversial and would support including it in
policy now.
I don't think it needs seconds because it is non-normative.

(As an aside, reading the summary, I expected to find the patch
something I was not entirely happy with.  I was planning to hold my
nose, and neither support the patch nor object.  However, since you
brought it up again, I read the full patch and find I like the patch a
lot better than your summary of it:-)

Luca Boccassi <bluca@debian.org> writes:

I believe it would be an error to move Policy away from explicitly saying
/bin/sh as long as we have unmerged systems.  For as long as we have to
support unmerged systems, which includes the buildds because quite a
surprising range of packages end up needing to be installed on buildds
during package builds, packages must use /bin/sh and may not use
/usr/bin/sh.

This is not currently explicit in Policy because previously it wasn't
necessary.  Packages using /usr/bin/sh would simply not work, thus forcing
the issue.  But right now, if we were writing Policy from scratch, we
would need to explicitly say that using /bin/sh is a must in order to
avoid breaking the buildds, since /usr/bin/sh would appear to work locally
and then potentially cause weird problems during package builds.  (Ideally
build failure, but a lot of strange things can happen when paths are
missing during a build.)

Presumably this is getting fixed in short order, so it's not worth the
intermediate Policy change to add that language only to remove it again.
But similarly, I don't think it's correct to relax Policy language about
the /bin/sh path for as long as using /usr/bin/sh is potentially a
release-critical bug.

Obviously this all becomes moot as soon as the buildds are /usr-merged.

The reason why this is not strange to me (which is not to say that this is
my personal preference) is that previously we had a rule requiring /bin/sh
enforced by a much harsher mechanism than Policy:  using /usr/bin/sh would
simply break.  So we *did* have a de facto rule, which we implicitly
dropped by doing /usr-merge, and the debate is whether to replace it with
a de jure rule.

Ansgar <ansgar@43-1.org> writes:

This is not a correct summary of Ian's proposal.  In the message that you
linked, Ian says:

    /bin and /lib etc. remain directories (so there is no aliasing).  All
    actual files are shipped in /usr.  / contains compatibility symlinks
    pointing into /usr, for those files/APIs/programs where this is needed
    (which is far from all of them).  Eventualloy, over time, the set of
    compatibility links is reduced to a mere handful.

I am absolutely certain that Ian would consider /bin/sh to be one of the
programs for which a compatibility symlink is needed, and one of the
remaining handful of links that would exist indefinitely into the future.
Indeed, he mentions /bin/sh explicitly later in that message.

Given that, I believe Ian's proposal is orthogonal to this bug.  For
/bin/sh and /usr/bin/sh, it would create the same aliasing and thus would
create the same question about how to talk about those paths in Policy.  I
therefore don't think resolution of this bug blocks on the TC bug.

Sam Hartman <hartmans@debian.org> writes:

To be clear, I personally agree with this, and I am sure that I will use
/bin/sh until the end of time, even if every extant UNIX system decides to
create a /usr/bin/sh alias for it (one way or the other).

But from an interface perspective, I think it adds some clarity to think
of it this way: When people started merging /bin and /usr/bin, they added
a new /usr/bin/sh interface for invoking the Bourne shell.  They may not
have intended to do this, but that's what they did, and there was nothing
inherent in the system to prevent people from using that interface.  So
people started doing so.

I'm sure that I'm not the only one who has encountered #!/usr/bin/sh
scripts in the wild, usually because the person who wrote the script was
using some Red Hat or Fedora distribution and didn't really think about
it.  Either they were used to the /usr/bin pattern from Perl and Python or
they ran "which sh" and, as Bill points out, that returns /usr/bin/sh,
etc.  Up until now, those scripts didn't work on Debian systems, which
caused some minor annoyances.  Now they do.

What has essentially happened given the number of distributions that have
done /usr-merge is that Linux has added /usr/bin/sh as a new interface
that is interchangeable with /bin/sh, and now nearly every current Linux
system you run across will also support that interface.  This is annoying
for UNIXes that don't support that interface (I don't know what the BSDs
are doing, for instance), but it's fairly invisible to users unless they
care about portability to non-Linux systems.  I care about portability to
non-Linux systems in at least some situations, but a lot of people don't.

(Incidentally, this interface argument is also why I have some nit-picks
about the wording of Luca's proposed patch, because the guarantees in
Policy apply specifically to /bin/sh and /usr/bin/sh, not to, say,
/usr/local/bin/sh or /usr/xpg4/bin/sh or whatever.  This is an admittedly
very minor point.)

I agree with this.  If we write example scripts, I think they should use
#!/bin/sh.  I don't think my (b) proposal is arguing against that; (b)
says that both paths work and you can use either of them, and Policy will
chose to use /bin/sh at least for the time being because, well, I'm one of
the Policy Editors and I care about things like that.  :)  I think Policy
would only use /usr/bin/sh in documentation and examples if we adopted
(c), which based on the discussion so far seems unlikely.
think it's what people are really asking and what directly addresses the
concern about bug reports asking maintainers to switch from /usr/bin/sh to
/bin/sh.  If Debian is going to contain scripts that use /usr/bin/sh, this
is a change and we should say so explicitly.

Bill pointed out precisely the way that I expect this to happen: a lot of
packages use PATH searches in their build systems to find common tools,
and although I think this is ill-advised, some of them do that for sh
(mostly because of old Solaris breakage).  As soon as the buildds are
/usr-merged, those packages are going to start using /usr/bin/sh instead
of /bin/sh because it's first in the PATH, assuming the buildds use the
same PATH setup as /etc/login.defs (I don't know if this is the case).
And the question is do we tell maintainers that they should fix this, or
do we accept that it will happen and not worry about it?

I personally probably would find a way to force /bin/sh somewhere because
I am a perfectionist, but it requires overriding the upstream build system
and maintaining that code going forward and a lot of maintainers aren't
going to want to bother.  I think Policy should provide some guidance to
both maintainers and bug reporters here about what we expect.

Precisely.  To me, that's the heart of the (b) position, although people
advocating for (b) are going to vary from "prefer /bin" to "prefer
/usr/bin" and (b) is sort of inherently a compromise position about what
we're going to *do* among people who have different preferences about
style.

Yeah, it sounds like you're thinking of exactly the case I was thinking
of.

I think this is the only part of your proposal that I have qualms about,
because I agree with Luca that people would use such encouragement to file
bugs about it, and I'm not sure we want to encourage those bugs.  It may
not matter a lot because the bugs will probably exist anyway, but I know
that putting things in Policy can sometimes raise the temperature of even
minor bugs.

I'd be happy to make some sort of statement in the section about shell
scripts that /bin/sh and /bin/bash are the traditional paths and will work
even if the script is copied to some non-Debian system, but I'm reluctant
to issue Policy advice that one therefore should use those paths.  But I
could be convinced as long as it's just advice, not a should.  (But,
again, Policy is not a style guide.)

I agree with this and I'm not seeing any disagreement in the discussion so
far.

This is the thing that I care the most about and would like to say
explicitly somewhere in Policy, even though that's beyond the scope of
Luca's original report.

I don't think Policy says anything about /usr-merge at all right now, and
once the buildds are merged and all Debian systems relevant to unstable
development are /usr-merged, we probably should.

Bill Allombert <ballombe@debian.org> writes:

I believe the reproducible build problem specifically will be largely
fixed by /usr-merging the buildds so that they look like all other Debian
systems.  I suspect the problems that you ran into in the past were
precisely because some systems on which the package was built were
/usr-merged and others were not.  But you make a good point that just
because the /bin and /usr/bin paths both work does not mean that package
build systems can pick randomly between them, since that undermines build
reproducibility.  They need to pick one or the other consistently.

I do think packages should be allowed to do a PATH search, and it's up to
the people doing a reproducible build to make sure the PATH stays
consistent from one build to the next.

Hi,

But the subject of this issue talks about "script interpreters", not
just `/bin/sh` (which I guess is safe to assume would be one of the
"handful").

It is unclear what files the Jackson symlink farm proposal would leave
in /bin.  Would /bin/python3 stay?  Or would it stay first, but dropped
at some later point?  What about /bin/perl, /bin/zsh, /bin/env, ...?

As far as I understand most compat symlinks should eventually be
dropped (but which is unclear). Therefore doing (c) is safe, but
anything else might use symlinks that eventually disappear.

(And really: what about calling /sbin/ifconfig, ... as well?  Here (c)
is again the only safe solution with the Jackson symlink farming
proposal.)

It was asked to clarify the plan for this multiple times, sadly the
proposers of the newly proposed filesystem layout have refused to give
any answer so far.

Ansgar

/bin/perl, /bin/env, /bin/python3 did not exist in the old scheme, so there is
no point in creating them now.

/bin/zsh and /bin/sed existed, though.

Cheers,

No, in Debian's current filesystem layout (Debian stable and later;
partly Debian oldoldstable and later) all of these exist.  They are
also used, both by software shipping in Debian and outside.

Dropping them would break existing software.  I'm assuming the Jackson
symlink proposal would retain them for this reason instead of breaking
software for no good reason.

Ansgar

Ansgar <ansgar@43-1.org> writes:

Oh, okay, I see what you're syaing now.  This is a bit beyond the scope of
the areas of Policy touched by Luca's proposal, but I can see why it would
indeed arise under Ian's proposal.  We've introduced new /bin interfaces
for every binary in /usr/bin, and if we went down that path we'd remove
most of those interfaces again.  That is indeed an argument for (c) for
*most* things, just not the areas touched by this diff (with the possible
exception of /bin/csh; I'm not sure if that would qualify for an exception
or not, these days).

So yes, I agree that the resolution of this bug would significantly affect
what we want to say in Policy about /usr-merge in general, even if not
what we say about /bin/sh.

I still don't feel like we need to wait for the TC bug to be resolved,
since there is a standing TC decision to make /bin a symlink to /usr/bin
and we can always change our wording later if that decision changes, but
we need to wait for the buildd /usr-merge anyway, so either way I don't
think we're ready to merge patches for this bug right now.