The new io_uring_spawn mechanism for spawning processes without forking
should be more efficient than fork+exec, especially when starting small
processes from large processes. Also posix_spawn and vfork+exec exist.

https://lwn.net/Articles/908268/

I think the order of preference for spawning processes should be:

 * io_uring_spawn: this is Linux-only and only in new versions. Prefer
   this over posix_spawn in case of an old glibc and new Linux kernel.
 * posix_spawn: this uses the appropriate mechanisms on each platform,
   glibc might be changing this to use io_uring_spawn where possible.
 * vfork+exec: this is similar to what glibc does for posix_spawn.
 * fork+exec: the current setup.

What I've heard of this sounds good, but as far as I can tell this is
not in upstream Linux, there are no patches anywhere to be found, no API
documentation, and the only references to the feature I can find
anywhere in a web search are all references to this one presentation
with no further detail.  It's of course possible that I've missed
something, but from what I can see it's far too early to even be able to
decide whether this would be usable, never mind being able to make use
of it on real systems.

I can see a few limitations here:

 * The standard API offers no way to set the working directory of the
   child process, which would be needed for pipecmd_chdir and
   pipecmd_fchdir.  However, glibc 2.29 added
   posix_spawn_file_actions_addchdir_np and
   posix_spawn_file_actions_addfchdir_np as GNU extensions.

 * I'm not totally sure how to translate pipecmd_nice into
   posix_spawn-speak; the documentation is, uh, opaque.  It's probably
   possible.

 * This wouldn't be usable for pipeline commands created using
   pipecmd_new_sequence, as posix_spawn isn't guaranteed to be
   async-signal-safe so can't be called between fork and exec, unlike
   fork.

However, we could always just restrict the conditions under which
posix_spawn is used, much as GLib's g_spawn_* functions do.  None of the
above features are used on mandb's hot path, for instance.

If somebody were to present me with a patch for this then I suppose I
might at least consider it (though with a healthy amount of
scepticism!); but it's difficult, and I'm not sure I have the necessary
skills to review it properly.  glibc's posix_spawn implementation has
this moderately fearsome comment at the top:

/* The Linux implementation of posix_spawn{p} uses the clone syscall directly
   with CLONE_VM and CLONE_VFORK flags and an allocated stack.  The new stack
   and start function solves most the vfork limitation (possible parent
   clobber due stack spilling). The remaining issue are:

   1. That no signal handlers must run in child context, to avoid corrupting
      parent's state.
   2. The parent must ensure child's stack freeing.
   3. Child must synchronize with parent to enforce 2. and to possible
      return execv issues.

   The first issue is solved by blocking all signals in child, even
   the NPTL-internal ones (SIGCANCEL and SIGSETXID).  The second and
   third issue is done by a stack allocation in parent, and by using a
   field in struct spawn_args where the child can write an error
   code. CLONE_VFORK ensures that the parent does not run until the
   child has either exec'ed successfully or exited.  */

Do I really want that complexity in libpipeline?  I'm not sure that I
do.  It's certainly not close to being a drop-in replacement for fork.
posix_spawn, maybe with GNU extensions, looks like a more appealing
option.

On looking at this further, there is another more serious problem with
using posix_spawn, despite it looking initially appealing:
pipeline_install_post_fork and pipecmd_pre_exec are used heavily by
man-db, and I don't see a way to implement either using posix_spawn
today.

The uses of pipeline_install_post_fork are mainly cleanup handlers that
could perhaps mostly be avoided (though it's not clear how libpipeline
could know that).  However, we use pipecmd_pre_exec to load a seccomp
filter and in one case to drop privileges.  Privilege-dropping could
perhaps be done using the POSIX_SPAWN_RESETIDS flag instead, although
we'd likely need to add some way to request that explicitly via
libpipeline.  But I see no way to implement loading a seccomp filter for
a child process using posix_spawn.

If there were some way to load a seccomp filter without needing to run
extra code between (v)fork and exec, then that would probably allow
solving this problem.  As it stands, though, I think this means that
posix_spawn can't in practice be used by man-db, and that means that I
have little interest in implementing it in libpipeline.

(Well, I suppose it might be possible to use a helper binary to load the
seccomp filter.  But I suspect that this would erase any performance
gains from doing the fork-and-exec more efficiently.)

A shame!

Sorry for the noise, seems like this request was a bit half-baked.

Thanks for looking into the proposals, it is unfortunate that they
aren't viable (yet). I'll leave it to you to close or wontfix the bug
or keep it open until io_uring_spawn becomes available eventually.