Package: dash
Version: 0.5.4-1
Severity: wishlist

When you do "bash -c foo", where foo is a simple command, it's execed.
It'd be nice if dash had this same optimisation (which should be
rather simpler than it is for bash).

It needn't even be done precisely, i.e. for all cases where it could
be used, as long as it covers the most common case:

sh -c foo

In general I believe that simply searching for the absence of a few
special characters `|&(){}' is a good way of catching cases that can
be optimised, without of course catching all of them.

The other alternative is to study how bash performs the same
optimisation.

If this sounds interesting, I'd be happy to try preparing a patch.

If this is ever implemented, care should be taken not to break backwards
compatibility. E.g.

$ bash -c 'readlink /proc/$$/exe'
/bin/readlink
$ bash -c ':; readlink /proc/$$/exe'
/bin/bash
$ dash -c 'readlink /proc/$$/exe'
/bin/dash

Cheers,

tags 436466 + upstream
quit

Hi Raphael,

Raphael Geissert wrote:

Anyway, it is academic to discuss that here.  Reuben, if you want this
to happen, please email dash@vger.kernel.org with a patch (being sure
to mention the change in observable behavior).

Regards,
Jonathan

Thanks for the direction. I am unsure why this particular "change in
observable behaviour" would be relevant, as it does not seem to
involve POSIX-specified behaviour, and dash is meant to be a
POSIX-compliant shell, no more, no less. If the behaviour quoted is
POSIX-mandated, then it would be a bug to change it; if it is not
POSIX-mandated, then it's a bug to rely on it.

Thanks for the direction. I am unsure why this particular "change in
observable behaviour" would be relevant, as it does not seem to
involve POSIX-specified behaviour, and dash is meant to be a
POSIX-compliant shell, no more, no less. If the behaviour quoted is
POSIX-mandated, then it would be a bug to change it; if it is not
POSIX-mandated, then it's a bug to rely on it.

I'm tempted to be snarky and say "you think this is just an
optimization, but it is not", but that's probably a bit too harsh.

I would tend not to want this type of hidden semantic change to occur
in dash if it can practically be avoided, because it would make dash
less simple and clean in an unobvious way in exchange for a small gain
in performance and a small gain in compatibility with Bash.

"exec foo ..." and "foo ..." do not have exactly equivalent semantics
even when each is the only command that will be executed by the shell
before exiting, and Bash's behavior is something I would consider a
somewhat kludgy extension behavior.  It is entirely possible for a
parent process that knows it is invoking a POSIX shell and needs the
process tree to be shaped a certain way to specify the 'exec'
itself---and for that matter it may as well just _do_ the exec itself
in most cases.

Depending on how it's implemented, this can also cause a sort of
inconsistency fault line.  Seemingly unrelated elements, such as
changing one parameter into a backquote expansion or adding a
redirect, can then introduce a new process even when the final
execve() could theoretically be done the same way.  Inserting, say, a
debugging echo before the actual command is virtually guaranteed to
not exec the final command directly.  Essentially the question boils
down to whether a provably final execution in a shell script should be
treated as a tail execution in the absence of an explicit 'exec'.  I
would argue that for cleanliness either they should always be (which
would be a lot of work and potentially more inconsistent with classic
shell behavior and may make certain operations very difficult) or they
should never be (which I believe is the current behavior).

I believe Tim Connors's notion that not doing the exec transformation
"causes the PPid line in /proc/$pid/status to be WRONG" is _thoroughly
unjustified_ from a theoretical perspective, since system() does not
specify that it will optimize away the shell process for a single
command.  This concern may however be valid as a matter of unfortunate
backward compatibility for many forms of GNU/Linux.

Nonetheless, POSIX:2001[1] seems to be ambiguous on the matter: it
states that "the shell shall execute the utility in a separate utility
environment with actions equivalent to calling the execve() function
defined in the System Interfaces volume of IEEE Std 1003.1-2001 [...]"
but the relation between utility environments and process environments
seems to be unspecified, so the exec transformation may be acceptable
to POSIX.  (The dash man page doesn't seem to say which version of
POSIX it's targeting.)

[1] http://www.opengroup.org/onlinepubs/000095399/utilities/xcu_chap02.html

Hi Drake,

Drake Wilson wrote:
[...]

Actually, let me take that back.  If you actually want a guarantee
that future versions of dash will have or lack this feature, you
can work with upstream[1] or the Austin common standards revision
group[2].

Hints:
. Since ksh93 (like ksh88, iirc) and bash already support this
  optimization and, well, it's been good for them, POSIX is not likely
  to change to forbid it.

. Goals for dash include code size, speed, POSIX conformance,
  simplicity, compatibility.  Patches with clear justification are
  much better received than arguments without patches (though bug
  reports are always appreciated).

Needless to say, a change in behavior is not going to happen in time
for squeeze regardless.

Thanks for your interest,
Jonathan

[1] dash@vger.kernel.org
[2] http://www.opengroup.org/austin/

Quoth Jonathan Nieder <jrnieder@gmail.com>, on 2010-11-26 01:10:47 -0600:

I wasn't previously aware of the ksh93 behavior.  So they _do_ do that
in cases where the entire script is not a single simple command.  (My
previous experience was always otherwise, including with Bash, pdksh,
I think zsh though that appears to exhibit ksh93-esque behavior now so
I may have misremembered, and one or two other shells.)

ksh93 appears to also handle various other tail positions, including
subshells and && and ||.  That's more the sort of thing I could get
behind if I were made Dictator of Shells.  :-)

Obviously my information above was out of date.  I amend it to say "in
many cases will not exec the final command directly".  (This means
that a program that wishes to be portable to multiple underlying
shells can rely even less on the process tree shape, so my previous
points mostly stand anyway.)

Let me be a little clearer, at the risk of restating myself.  I don't
personally need a guarantee either way, especially since it's
something I'll have to deal with regardless of what dash does in the
future.  I mostly wanted to ensure that people reading the bug trail
were aware of some of the subtler ramifications of a decision to do
tail execution optimization in shells, and in particular, I wanted to:

  - Add a dissenting voice to some ideas from upthread that making the
    last command a child of the shell is "wrong" and that relying on a
    highly compact process tree shape that was not actually requested
    is a sane thing to do in a POSIX or GNU/Linux environment, which
    IMHO it is not.

  - Point out that full tail execution optimization is not as
    localized as the "anything without shell meta-stuff" that some
    people think it is, and that partial tail execution optimization
    can lead to subtle problems if not done carefully.  Cf. a vaguely
    similar case where I was attempting to use an XSI shell builtin
    (which I was willing to rely on on the target system) as a single
    command in a series of commands in a makefile, and GNU Make
    decided that it didn't look enough like a shell thing and could be
    "optimized" into an exec, which naturally hosed the command.  Of
    course, the case of doing exec transformation _in_ the shell is
    not as bad as that.

Thanks for the attention.

Drake Wilson wrote:
operating system, possibly combined with an overzealous skipping of the
shell by gmake. It is rather different from the fork/exec problem with
sh -c.

POSIX requires that all standard utilities except special builtins be
available via the exec functions and the utilities that can invoke
user-specified other utilities (env, find, nice, nohup, time, xargs).

In SUSv4, the special builtins are . (dot), : (colon), break, continue,
eval, exec, exit, export, readonly, return, set, shift, times, trap and
unset; none of these are under the XSI option. An implementation is not
required to provide them as external programs (and there seems no point
in doing so). Gmake must know about these and not try to exec them
itself.

Other builtins that do not exist as external programs yet need to be
provided. An easy way is to link a shell script like the below to
/usr/bin/alias, /usr/bin/bg, /usr/bin/cd, /usr/bin/command, etc.:

#!/bin/sh
command -p ${0##*/} "$@"

The 'command -p' serves to avoid looping when the PATH is set to a value
like /usr/bin:%builtin which is an undocumented way to prefer
/usr/bin/FOO to a builtin command FOO. The 'command' utility itself is
not subject to this because dash implements the POSIX requirement that
various (usually builtin) utilities such as 'command' cannot be
overridden via PATH (but unlike the special builtins can be overridden
with a function).

Regarding sh -c optimization, I am in favour of this. Uselessly waiting
'sh -c' processes annoy me. I made the change for FreeBSD 8.0 sh, which
is very similar to dash. The SVN changeset is r194128. The change
appears to work for dash as well.

tags 436466 + patch
forwarded 436466 http://thread.gmane.org/gmane.comp.shells.dash/556
quit

Jilles Tjoelker wrote:

Thanks.

The only concern I have with this change is the handling of $PPID

$ cat /tmp/t
echo $PPID
$ sh -c /tmp/t
17254
$ sh -c 'exec /tmp/t'
13318
$

Would this be a regression and, if yes, an acceptable one?

Regards, Gerrit.

Hi Gerrit,

It's a noticeable behavior change and so it should be documented in
the changelog, I agree.

I think the change would be acceptable.  Many other shells (e.g.,
ksh93, bash) follow the same rule so it doesn't hurt script
portability.  POSIX clearly allows it: when XCU 2.12 says

	Utilites other than the special built-ins [...]
	shall be invoked in a separate environment that consists of
	the following.

it is very careful to not include the process ID as part of the
definition.  Natural enough.  Typically non-special builtins like "cd"
are run in the same process, even while they get their own set of
variables, cwd, redirections, and so on.

Regards,
Jonathan

We believe that the bug you reported is fixed in the latest version of
dash, which is due to be installed in the Debian FTP archive:

ash_0.5.7-1_all.deb
  to main/d/dash/ash_0.5.7-1_all.deb
dash_0.5.7-1.diff.gz
  to main/d/dash/dash_0.5.7-1.diff.gz
dash_0.5.7-1.dsc
  to main/d/dash/dash_0.5.7-1.dsc
dash_0.5.7.orig.tar.gz
  to main/d/dash/dash_0.5.7.orig.tar.gz



A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 436466@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Jonathan Nieder <jrnieder@gmail.com> (supplier of updated dash package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)
Format: 1.8
Date: Wed, 17 Aug 2011 00:30:30 -0500
Source: dash
Binary: dash ash
Architecture: all source
Version: 0.5.7-1
Distribution: unstable
Urgency: low
Maintainer: Gerrit Pape <pape@smarden.org>
Changed-By: Jonathan Nieder <jrnieder@gmail.com>
Description:
 ash        - compatibility package for dash
 dash       - POSIX-compliant shell
Closes: 436466 539909 556521 582952 611556 624172 624715
Changes:
 dash (0.5.7-1) unstable; urgency=low
 .
   * new upstream release.
     * optimize dash -c "script" to notice the last command and
       exec it without forking (thx Jilles Tjoelker; closes:
       #436466).  This change is inspired by a similar trick in
       ksh, bash, and zsh and can be useful in commands passed to
       system(), popen(), and Makefiles, where "exec" is usually
       not explicitly used.
     * remove spurious space in descriptions of PS1, PS2, PS4 (thx
       Kalle Olavi Niemitalo; closes: #624715).
     * test builtin: use faccessat if available (closes: #539909,
       #556521) so ACLs and other corner cases can be handled
       correctly.
   * debian/diff/:
     - remove patches applied upstream (i.e. all except 0045, 0046).
     - 0045-SHELL-print-n-upon-EOF-...diff: rename to 0001-*.
     - 0046-PARSER-Remove-backslash-...diff: rename to 0002-*.
     - 0003-VAR-Disable-LINENO-support.diff: new; remove LINENO
       support (closes: #582952, reopens: #540685).
   * debian/po/sk.po: [INTL:sk] Slovak debconf templates (thx
     Slavko; closes: #611556).
   * debian/po/nl.po: [INTL:nl] Dutch debconf templates (thx
     Jeroen Schot; closes: #624172).
Checksums-Sha1:
 a9e509c1fb5691909c855ccfa23a8591341a31db 1013 dash_0.5.7-1.dsc
 a3ebc16f2e2c7ae8adf64e5e62ae3dcb631717c6 223794 dash_0.5.7.orig.tar.gz
 3901d3762aff8fbad117707c6d89e022ccb48944 39151 dash_0.5.7-1.diff.gz
 8691b607c0ee812f3ce909adfa8707d839747410 28692 ash_0.5.7-1_all.deb
Checksums-Sha256:
 19bd38f4dd40113bfb3c4af85a67db2f612acad01250e956a640d541e3ff9485 1013 dash_0.5.7-1.dsc
 ae89fa9f1145b7748cf0740e1df04cd52fdf8a285da4911dd0f04983efba4e39 223794 dash_0.5.7.orig.tar.gz
 d928ac4862dcea1b796a517709d3a102e23f812c14e9e6f9dc3d3ad7d4d7b6f4 39151 dash_0.5.7-1.diff.gz
 90f7ba2d41deef129826bbf4beb88523484b7f6e68c651068a8bef978d72d1a6 28692 ash_0.5.7-1_all.deb
Files:
 5ded4af64a7fb025e170cd1e60b46ef6 1013 shells optional dash_0.5.7-1.dsc
 f6cedb10ae7258adb5ab17a10ae80d51 223794 shells optional dash_0.5.7.orig.tar.gz
 7069e2f8d4784764829a593e5c364301 39151 shells optional dash_0.5.7-1.diff.gz
 195a697f0cfbe1c9e5181b68708aba02 28692 shells optional ash_0.5.7-1_all.deb
iEYEARECAAYFAk5OXIAACgkQGJoyQbxwpv/o+ACgpPF76mAQsz8nWJimGWgfZCxP
8SUAnRq4g6I6rIOGiipv1YYOTZ7YOXL+
=LdLj
-----END PGP SIGNATURE-----

We believe that the bug you reported is fixed in the latest version of
dash, which is due to be installed in the Debian FTP archive:

ash_0.5.7-1_all.deb
  to main/d/dash/ash_0.5.7-1_all.deb
dash_0.5.7-1.diff.gz
  to main/d/dash/dash_0.5.7-1.diff.gz
dash_0.5.7-1.dsc
  to main/d/dash/dash_0.5.7-1.dsc
dash_0.5.7.orig.tar.gz
  to main/d/dash/dash_0.5.7.orig.tar.gz



A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 436466@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Jonathan Nieder <jrnieder@gmail.com> (supplier of updated dash package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)
Format: 1.8
Date: Wed, 17 Aug 2011 00:30:30 -0500
Source: dash
Binary: dash ash
Architecture: all source
Version: 0.5.7-1
Distribution: unstable
Urgency: low
Maintainer: Gerrit Pape <pape@smarden.org>
Changed-By: Jonathan Nieder <jrnieder@gmail.com>
Description:
 ash        - compatibility package for dash
 dash       - POSIX-compliant shell
Closes: 436466 539909 556521 582952 611556 624172 624715
Changes:
 dash (0.5.7-1) unstable; urgency=low
 .
   * new upstream release.
     * optimize dash -c "script" to notice the last command and
       exec it without forking (thx Jilles Tjoelker; closes:
       #436466).  This change is inspired by a similar trick in
       ksh, bash, and zsh and can be useful in commands passed to
       system(), popen(), and Makefiles, where "exec" is usually
       not explicitly used.
     * remove spurious space in descriptions of PS1, PS2, PS4 (thx
       Kalle Olavi Niemitalo; closes: #624715).
     * test builtin: use faccessat if available (closes: #539909,
       #556521) so ACLs and other corner cases can be handled
       correctly.
   * debian/diff/:
     - remove patches applied upstream (i.e. all except 0045, 0046).
     - 0045-SHELL-print-n-upon-EOF-...diff: rename to 0001-*.
     - 0046-PARSER-Remove-backslash-...diff: rename to 0002-*.
     - 0003-VAR-Disable-LINENO-support.diff: new; remove LINENO
       support (closes: #582952, reopens: #540685).
   * debian/po/sk.po: [INTL:sk] Slovak debconf templates (thx
     Slavko; closes: #611556).
   * debian/po/nl.po: [INTL:nl] Dutch debconf templates (thx
     Jeroen Schot; closes: #624172).
Checksums-Sha1:
 a9e509c1fb5691909c855ccfa23a8591341a31db 1013 dash_0.5.7-1.dsc
 a3ebc16f2e2c7ae8adf64e5e62ae3dcb631717c6 223794 dash_0.5.7.orig.tar.gz
 3901d3762aff8fbad117707c6d89e022ccb48944 39151 dash_0.5.7-1.diff.gz
 8691b607c0ee812f3ce909adfa8707d839747410 28692 ash_0.5.7-1_all.deb
Checksums-Sha256:
 19bd38f4dd40113bfb3c4af85a67db2f612acad01250e956a640d541e3ff9485 1013 dash_0.5.7-1.dsc
 ae89fa9f1145b7748cf0740e1df04cd52fdf8a285da4911dd0f04983efba4e39 223794 dash_0.5.7.orig.tar.gz
 d928ac4862dcea1b796a517709d3a102e23f812c14e9e6f9dc3d3ad7d4d7b6f4 39151 dash_0.5.7-1.diff.gz
 90f7ba2d41deef129826bbf4beb88523484b7f6e68c651068a8bef978d72d1a6 28692 ash_0.5.7-1_all.deb
Files:
 5ded4af64a7fb025e170cd1e60b46ef6 1013 shells optional dash_0.5.7-1.dsc
 f6cedb10ae7258adb5ab17a10ae80d51 223794 shells optional dash_0.5.7.orig.tar.gz
 7069e2f8d4784764829a593e5c364301 39151 shells optional dash_0.5.7-1.diff.gz
 195a697f0cfbe1c9e5181b68708aba02 28692 shells optional ash_0.5.7-1_all.deb
iEYEARECAAYFAk5OXIAACgkQGJoyQbxwpv/o+ACgpPF76mAQsz8nWJimGWgfZCxP
8SUAnRq4g6I6rIOGiipv1YYOTZ7YOXL+
=LdLj
-----END PGP SIGNATURE-----

unarchive 436466
reopen 436466 !
thanks

Because of bug #642922 (archived, but has some interesting discussion),
this patch for this bug ended up being backed out.

Hopefully this bug won't be forgotten by someone who understands the
problems in 642922, which seem to have been fixed elsewhere.

tags 642835 = upstream
block 436466 by 642835
tags 436466 + fixed-upstream
fixed 436466 dash/0.5.7-1
found 436466 dash/0.5.7-2
quit

Tim Connors wrote:

Thanks for the reminder, Tim.  Indeed, this won't be forgotten.

Looks it had been forgotten :)

I gather that the ocamlbuild bug has been fixed so can this be
reinstated please?

Thanks,

Herbert Xu wrote:
[...]
<https://github.com/ocaml/ocamlbuild/issues/164>, but
https://github.com/ocaml/ocamlbuild/commit/00e05f686e15b07ac4268fcd10cf3738a5c28467
was applied with the proposed fix in ocamlbuild 0.9.0.

I suspect this means that the ocamlbuild bug was indeed fixed, and
that we should be able to reinstate the patch.

Except...  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=642835;msg=55
says the bug is present in ocamlbuild 0.10.1.  Ximin, do you remember
where that version number came from?  Is it possible that the bug is
*not* present in that version?

Thanks,
Jonathan

 ❦ 27 May 2020 08:42 -07, Jonathan Nieder:

Once bullseye is released, would it be possible to reinstate the patch?
Or maybe in experimental. This way, it will be easier to check if the
problem is still present. Moreover, everyone should not carry the burden
of failures from ocamlbuild to correctly track processes. Another
workaround would be to use bash for ocamlbuild if the problem is still
here.

While the footprint of /bin/sh is small, it is not zero and since most
shells optimize this use case, many programs assume they can spawn
process through `/bin/sh -c` at no cost. In my case, I was curious why
processes spawned by i3 and its ecosystem were all leaving a /bin/sh
instance behind.