By default zsh loads completions from /usr/share/zsh/site-functions and while the name of that is perhaps not great, as it's not site-specific[1], it's a standard location. Debian has overridden this, so zsh does not look there, but instead in /usr/share/zsh/vendor-functions and /usr/share/zsh/vendor-completions This means that, as a developer of software that has zsh completions, I first had to learn that the Debian location, which I had made my software's Makefile install to, is non-standard (which took a while since I use Debian everywhere). Then I had to grapple with the question of how to deal with this inconsistency; should I try to probe if it's a Debian system or what? For me this involved looking at how several other Debian packages that ship files in /usr/share/zsh/vendor-completions got them there. I learned that few upstream software packages bother to probe for the correct location, the one I found that does probe does so in a way that would not work if building the software with zsh not installed. I concluded a build system option and documentation was the sanest approach for my software. The Debian package of my software will need to use that in its debian/rules. The rationalle for the change in #620452 seems too weak to justify this added complexity. zsh could at least continue looking in the default location as well as whatever other locations Debian wants to patch in.
Joey Hess wrote on Fri, 16 Aug 2019 18:35 +00:00:
...
Let's see:
$ /usr/bin/zsh -fc 'print -rl $fpath' | head -3 # that's the Debian package, 5.7.1-1
/usr/local/share/zsh/site-functions
/usr/share/zsh/vendor-functions
/usr/share/zsh/vendor-completions
$ /srv/zsh/bin/zsh -fc 'print -rl $fpath' | head -3 # that's zsh compiled from upstream with --prefix=/srv/zsh
/usr/local/share/zsh/site-functions
/srv/zsh/share/zsh/site-functions
/srv/zsh/share/zsh/5.6.2-test-2/functions
$
So /usr/local/share/zsh/site-functions is still there for stuff installed
by the administrator, and vendor-functions and vendor-completions are
added in there for stuff provided by other Debian packages. Are you
just asking to re-add ${PREFIX:-/usr}/share/zsh/site-functions there,
presumably between the first and second elements?
On the one hand, it _would_ be consistent with upstream, but on the
other hand, nobody should be putting anything in a site-functions
directory under /usr/share (admins should be putting their
customizations in /usr/local/share; other Debian packages should be
putting their stuff in vendor-functions or vendor-completions). I don't
have a strong opinion on this, so I'll defer to others to make the call.
I suppose that if we add this, we should also add
${PREFIX}/share/zsh/${ZSH_VERSION}/functions, shouldn't we?
Cheers,
Daniel
be pragmatic for Debian's zsh to search /usr/share/zsh/site-functions.
It should be searched after /usr/local/share/zsh/site-functions, and at
a similar priority to vendor-functions and vendor-completions (I think
either immediately before or immediately after those would be fine).
zsh doesn't seem to provide a way for packages' build systems to discover
what directory they are meant to install completions into (unlike
bash-completion, which provides /usr/share/pkgconfig/bash-completion.pc
which can be queried for a completionsdir variable), so upstream source
code for packages that want to install zsh completions will typically
install to ${datadir}/zsh/site-functions by default. I don't think it
scales well to insist that every upstream package that wants to install
zsh completions should have a --with-zshcompletiondir configure option
(or equivalent), and that every Debian package that installs zsh
completions must explicitly override the installation directory in
d/rules or in d/*.install.
The simplest way to resolve this would be for Debian's zsh to search
/usr/share/zsh/site-functions as Joey requested. That way, packages with
zsh completions, such as bluez, would not need to do anything special.
I realise the Debian zsh maintainers want to use a directory with a less
misleading name, but it seems we currently have 26 packages in unstable
that get this wrong, and if I hadn't tested with zsh before uploading,
the next version of bubblewrap would have been the 27th:
$ apt-file search --filter-suites=unstable /usr/share/zsh/site-functions
arch-install-scripts: /usr/share/zsh/site-functions/_archinstallscripts
bluez: /usr/share/zsh/site-functions/_bluetoothctl
cpu-x: /usr/share/zsh/site-functions/_cpu-x
draai: /usr/share/zsh/site-functions/_draai
elogind: /usr/share/zsh/site-functions/_loginctl
flameshot: /usr/share/zsh/site-functions/_flameshot
flightgear: /usr/share/zsh/site-functions/_fgfs
foot: /usr/share/zsh/site-functions/_foot
foot: /usr/share/zsh/site-functions/_footclient
gallery-dl: /usr/share/zsh/site-functions/_gallery-dl
gpaste: /usr/share/zsh/site-functions/_gpaste-client
guix: /usr/share/zsh/site-functions/_guix
kdeconnect: /usr/share/zsh/site-functions/_kdeconnect
kickpass: /usr/share/zsh/site-functions/_kickpass
laminarc: /usr/share/zsh/site-functions/_laminarc
libinput-tools: /usr/share/zsh/site-functions/_libinput
lr: /usr/share/zsh/site-functions/_lr
nvme-cli: /usr/share/zsh/site-functions/_nvme
pcp: /usr/share/zsh/site-functions/_pcp
pencil2d: /usr/share/zsh/site-functions/_pencil2d
polybar: /usr/share/zsh/site-functions/_polybar
polybar: /usr/share/zsh/site-functions/_polybar_msg
profile-sync-daemon: /usr/share/zsh/site-functions/_psd
vifm: /usr/share/zsh/site-functions/_vifm
wlogout: /usr/share/zsh/site-functions/_wlogout
xe: /usr/share/zsh/site-functions/_xe
This is nearly 20% of the list of 138 packages in unstable that install
files into /usr/share/zsh, so if it's an error, then it is a common error.
https://bugs.launchpad.net/ubuntu/+source/borgbackup/+bug/1827232 is a good
example of a package that historically got this wrong and was subsequently
fixed, taking up various people's time in the process.
Or, if the Debian zsh maintainers' policy is that installing to
/usr/share/zsh/site-functions is always wrong, then we should have a
Lintian check that flags it as a packaging error and recommends using
vendor-completions or vendor-functions instead, and bug reports against
the 26 packages listed above (the only ones I have checked are bluez
and nvme-cli, for which I've confirmed that the zsh completions are
non-functional at the moment).
If there are other directories that are commonly used by upstream software
but are considered wrong in Debian, then those should be covered by the
same Lintian check.
It seems unlikely that third-party software will install into
${datadir}/zsh/${ZSH_VERSION}/functions, because it would have to go out
of its way to discover the current zsh version, and then would cease
to work whenever zsh is upgraded. So I don't think this is necessary
or desirable.
smcv
The simple fact is that as an upstream author who used the debian locations because they were the ones that worked on my system, I get bug reports from users of other systems that it's not right for wider uses of zsh. And Debian seems to be leaving it up to me to deal with it, which just makes me want to not include zsh completions, honestly. (I got another bug report about this today, for etckeeper.)
Joey Hess wrote on Sat, Jan 08, 2022 at 16:27:53 -0400:
The right thing to do *at this time* is fairly straightforward:
- Packages that are part of Debian should install to
/usr/share/zsh/vendor-*.
- Upstream packages have three options:
+ Install to /usr/local/share/zsh/site-functions (regardless of their
own $PREFIX _and_ of zsh's $PREFIX)
+ Install to ${PREFIX of the zsh build}/share/zsh/site-functions
+ Install to some other place and use a post-install message to direct
the user to amend their $fpath manually.
That's not to say that things can't be improved going forward. Of
course they can. I'm just trying to document the best practice at this
time.
Going forward, perhaps the vendor-* convention could be upstreamed, or
upstream could make it easier to discover its prefix, or /etc/zsh/zshrc.d/
could be added (there's a separate ticket for that but my quick grep
didn't find it), or…
Indeed, this ticket seems a bit open-ended. What's being asked here?
Just to add /usr/share/zsh/site-functions [sic] to Debian's zsh's
default $fpath? Or something more general, e.g., "Ensure there are as
few obstacles as possible to developers of third-party autoloadable
functions"?
Cheers,
Daniel
Hi Daniel, Daniel Shahaf wrote: The big question to me seems "how?". You probably had https://bugs.debian.org/776663 in mind which has been filed against zsh-common, not zsh (but src:zsh), so I suspect you haven't it found because of that. There's also an Ubuntu bug report on a similar topic at https://bugs.launchpad.net/ubuntu/+source/zsh/+bug/1800280 which is specifically about parsing /etc/profile, i.e. the same file bash parses. Ack! And JFTR: I actually don't have an opinion on this, either, as I'm too far away from knowing the non-debian conventions. I'd though would like to see a consensus inside the Debian Zsh team on how (and where) to go forward. I'd especially would be happy to hear about Frank's (Cc'ed) opinion as he and Daniel are those who are most clued about upstream things and especially upstream conventions. (And because I know that Frank has a quite clear opinion on #776663 — where I actually do have an opinion, too, albeit a different one than Frank.) Joey: I re-added the actual bug report title to the subject to make it clear about what topic the discussion is. Regards, Axel
Daniel Shahaf wrote: […] This is precisely the right fpath entry for site local function files. The Debian package set this to a /usr/local directory even when configu- red with --prefix=/usr since before the current team took over. This is the right thing to do in any case, since non-package files should just be dropped into /usr/share/. We *added* the ‘vendor-*’ directories to get fpath entries that are under the control of the package system for packages to drop additional functions into. It's been this way since about 2011. A couple of years later (2014-ish) zsh upstream changed its compile time configuration semantics with respect to site-function directories: It now ensures that /usr/local/share/zsh/site-functions is *always* there and on top of the fpath variable. This acknowledges that /usr/local/ is the right place for site local additions, which I don't think anybody disputes. The change in handling site-function directories is that compile time configuration also adds a PREFIX/share/zsh/site-functions directory, with lower priority than the one in /usr/local. The reason this doesn't get added on Debian right now is that we're still setting ‘--enable-site-fndir=/usr/local/share/zsh/site-functions’ explicitly. Dropping this setting would change this. Since Debian's ‘vendor-*’ directory handling predates this by years, changing this, effectively adding a second path for the same purpose seems inelegant, since it adds redundancy where none is needed. This however, sucks: % apt-file search usr/share/zsh/site-functions/ | wc -l 30 Because that is thirty functions that will not work. When we added the ‘vendor-*’ stuff, we filed bug reports for packages that tried this, because even back then, it wouldn't have worked, because site-functions always was in /usr/local with Debian's zsh. That resulted in this: % apt-file search vendor-completions | wc -l 166 I am not sure if there's an elegant way to resolve this, because the ‘vendor-*’ directories are the documented way for zsh packages to add functions for more than a decade. I don't think dropping them is a good idea, because it would break backward compatibility. And as I said, just adding the second --prefix based site-functions entry would litter the system by added multiple destinations for the same purpose. Maybe there's a way to add a lintian check for the installation path of zsh function files in Debian packages. With that, we could add the pre- fix based site-functions directory, but deprecate its use. That way, packages that disrespect the zsh package's policy still work, while keeping the possibility of a clean system, in case all package adhere to the policy. We could file a bunch of bugs for the packages that are cur- rently using /usr/share/zsh/site-functions right now. Regards, Frank
Hi Frank, thanks a lot for trying to wrap the history of all these directories! I wasn't aware of this non-obvious timeline with Debian having added vendor-* directories before upstream last updated their search paths. I see. I agree here. Ack. This is clearly the majority. (IMHO) Good. Good idea! I can work on this. Ack, that thought came to me also after having read your mail half-through. It though has the danger that some so far not active plugins will suddenly start to work. So we add least need to debian/zsh.bug-script. Currenty it only checks for packages that ship files in /usr/share/zsh/vendor-completions/ and /usr/share/zsh/vendor-functions/. Debian prefers lintian warnings over mass bug filing. Mass bug filing needs to be discussed on the debian-devel mailing list first. What I now still wonder to (hopefully) address Joey's issue: Is there a way to build the Debian zsh (respectively probably the zsh-dev) package in a way so that locally installed Zsh extensions get cleanly installed into /usr/local/ while Debian packages still install stuff to /usr/ (preferably these vendor-* directories)? I wonder if something like a dh_zsh helper could help here (i.e. for the vendor-* directory part) as well. Regards, Axel
Axel Beckert wrote:
[…]
True. Although "normal" function don't get activated automatically. Only
files that match ‘_*’ in $fpath get activated automatically by compinit,
if the user calls that function in their configuration files.
That is fair.
Well, I am not sure what "plugins" do in terms of installation proce-
dures. The only such package that I have written¹ ships a script that
asks you for a directory that it should use for deployment.
If I were to do some sort of automatic site-local installation, I'd pro-
bably install to this: zsh -c '${fpath[1]}'
I suppose we could make it do this:
Install files from the source directory, that are not generally instal-
led by the upstream installation process. I think that this is still
more often than not the case:
dh_zsh -c -s examples/_foobar contrib/_quux
dh_zsh -f -s examples/some-function
Similarly, move functions from the in the packaging root directory, in
case things get installed by the upstream installation procedure:
dh_zsh -c -p /usr/local/share/zsh/site-functions/_foobar
dh_zsh -f -p /usr/share/zsh/site-functions/some-function
The mnemonics here are:
-c → Completion system function; put into vendor-completions
-f → General, auto-loadable function file.
-s → From Source directory
-p → From package root directory.
I suppose we could also add a ‘-r’ option to make the process recursive.
In addition, we could add an automatic mode for stuff in the packaging
root directory, if called without arguments, which should fix naïve in-
stallation procedures automatically, due to zsh's convention about names
for completion functions:
- Scan these directories:
- /usr/local/share/zsh/site-functions
- /usr/share/zsh/site-functions
- Move files matching ‘_*’ to vendor-completions
- Move the other files to vendor-functions.
- Remove those two directories (they should be empty now)
I think that should support most installation schemes.
WDYT?
Regards, Frank
Axel Beckert wrote on Sun, Jan 09, 2022 at 08:52:04 +0100:
Indeed. Thanks. (I didn't pass «--source» to querybts(1).)
Well, if I were designing things from scratch, I'd probably go for
having packages in Debian install to /usr/share/zsh/foo whatever their
upstreams install by default to /usr/local/share/zsh/foo. That'd be
exactly analogous to Debian's semantics of /usr/bin v. /usr/local/bin
("owned by packages" v. "owned by the local sysadmin") and would result
in short, readable package build scripts (basically just the equivalent
of «./configure --prefix=/usr»).
However, /usr/share/zsh/vendor-* exist, and I see no reason to break
working code. I suppose we could deprecate these two dirs but not
actually drop support for them before zsh 6.0. In lintian terms,
I suppose that means a ≤info lintian check for zsh 5.x and a ≥warning
lintian check for zsh 6.x.
And for upstreams… well, that's a discussion I'd rather have upstream.
Cheers,
Daniel